[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:58:41 dist-test-slave-f1j0 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:58:41 dist-test-slave-f1j0 krb5kdc[395](info): set up 2 sockets
Jun 20 02:58:41 dist-test-slave-f1j0 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 02:58:42.612003 388 mini_kdc.cc:121] Time spent starting KDC: real 2.162s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 02:58:42.804874 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.192s user 0.000s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 02:58:42 dist-test-slave-f1j0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:58:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:58:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:58:42.947579 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36931
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:58:43.271593 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:58:43.271926 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:58:43.272079 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:58:43.279206 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:58:43.279351 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:58:43.279455 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:58:43.279546 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:58:43.279628 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:58:43.289709 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:58:43.291090 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:58:43.292416 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:58:43.304935 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:43.304937 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:43.304970 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:58:43.305900 408 server_base.cc:1034] running on GCE node
I20260620 02:58:43.306743 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:58:43.308190 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:58:43.309412 408 hybrid_clock.cc:648] HybridClock initialized: now 1781924323309324 us; error 101 us; skew 500 ppm
I20260620 02:58:43.309796 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:58:43 dist-test-slave-f1j0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:58:43.316483 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:58:43.318055 408 webserver.cc:460] Webserver started at http://127.0.97.62:38335/ using document root <none> and password file <none>
I20260620 02:58:43.318764 408 fs_manager.cc:359] Metadata directory not provided
I20260620 02:58:43.318903 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:58:43.319209 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:58:43.322310 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "0696b1e63594418c8bba508f10b4b53a"
format_stamp: "Formatted at 2026-06-20 02:58:43 on dist-test-slave-f1j0"
server_key: "cd7b0602ebb5cc2ed651a636e66bd13a"
server_key_iv: "aa97302dcc5e1358e29908cb0ea32d7f"
server_key_version: "encryptionkey@0"
I20260620 02:58:43.323074 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "0696b1e63594418c8bba508f10b4b53a"
format_stamp: "Formatted at 2026-06-20 02:58:43 on dist-test-slave-f1j0"
server_key: "cd7b0602ebb5cc2ed651a636e66bd13a"
server_key_iv: "aa97302dcc5e1358e29908cb0ea32d7f"
server_key_version: "encryptionkey@0"
I20260620 02:58:43.328821 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:58:43.332309 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:43.333645 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:58:43.333834 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "0696b1e63594418c8bba508f10b4b53a"
format_stamp: "Formatted at 2026-06-20 02:58:43 on dist-test-slave-f1j0"
server_key: "cd7b0602ebb5cc2ed651a636e66bd13a"
server_key_iv: "aa97302dcc5e1358e29908cb0ea32d7f"
server_key_version: "encryptionkey@0"
I20260620 02:58:43.334026 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:58:43.352105 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:58:43.356208 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:58:43.356508 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:58:43.375907 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36931
I20260620 02:58:43.375941 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36931 every 8 connection(s)
I20260620 02:58:43.377298 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:58:43.380975 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:43.383888 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 408
I20260620 02:58:43.384191 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 02:58:43.387039 388 external_mini_cluster.cc:1428] Setting key e7512c28c19fe604fc7b8c1ccc41fb10
I20260620 02:58:43.395160 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a: Bootstrap starting.
I20260620 02:58:43.398687 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:43.399680 475 log.cc:824] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a: Log is configured to *not* fsync() on all Append() calls
I20260620 02:58:43.402949 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a: No bootstrap required, opened a new log
I20260620 02:58:43.412150 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0696b1e63594418c8bba508f10b4b53a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36931 } }
I20260620 02:58:43.412604 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:43.412725 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0696b1e63594418c8bba508f10b4b53a, State: Initialized, Role: FOLLOWER
I20260620 02:58:43.413329 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0696b1e63594418c8bba508f10b4b53a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36931 } }
I20260620 02:58:43.413558 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:58:43.413754 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:58:43.413914 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:43.416074 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0696b1e63594418c8bba508f10b4b53a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36931 } }
I20260620 02:58:43.416643 475 leader_election.cc:302] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0696b1e63594418c8bba508f10b4b53a; no voters:
Jun 20 02:58:43 dist-test-slave-f1j0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:43.417349 475 leader_election.cc:288] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:58:43.417451 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:58:43.419554 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a [term 1 LEADER]: Becoming Leader. State: Replica: 0696b1e63594418c8bba508f10b4b53a, State: Running, Role: LEADER
I20260620 02:58:43.419966 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:58:43.420125 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0696b1e63594418c8bba508f10b4b53a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36931 } }
I20260620 02:58:43.423221 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0696b1e63594418c8bba508f10b4b53a. Latest consensus state: current_term: 1 leader_uuid: "0696b1e63594418c8bba508f10b4b53a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0696b1e63594418c8bba508f10b4b53a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36931 } } }
I20260620 02:58:43.423563 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a [sys.catalog]: This master's current role is: LEADER
I20260620 02:58:43.424111 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0696b1e63594418c8bba508f10b4b53a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0696b1e63594418c8bba508f10b4b53a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36931 } } }
I20260620 02:58:43.424363 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a [sys.catalog]: This master's current role is: LEADER
I20260620 02:58:43.433838 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:43.391724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52898 (local address 127.0.97.62:36931)
0620 02:58:43.392838 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:43.392862 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:43.394277 (+ 1415us) server_negotiation.cc:407] Connection header received
0620 02:58:43.403630 (+ 9353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:43.403659 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:43.404138 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:43.404554 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:43.407187 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:43.409702 (+ 2515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:43.411299 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:43.412292 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:43.412464 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:43.419080 (+ 6616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:43.419101 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:43.419129 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:43.419204 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:43.424259 (+ 5055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:43.425598 (+ 1339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:43.425612 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:43.425623 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:43.425886 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:43.426537 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:43.426557 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:43.426563 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:43.427403 (+ 840us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:43.427768 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:43.431531 (+ 3763us) server_negotiation.cc:299] Negotiation successful
0620 02:58:43.432082 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":431,"threads_started":1}
I20260620 02:58:43.436048 487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:58:43.441560 487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:58:43.450773 487 catalog_manager.cc:1347] Generated new cluster ID: 02e538748aed480795c35fb44023bf26
I20260620 02:58:43.450894 487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:58:43.458643 487 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:58:43.460445 487 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:58:43.475663 487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0696b1e63594418c8bba508f10b4b53a: Generated new TSK 0
I20260620 02:58:43.476796 487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:58:43.527683 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36931
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:58:43.859678 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:58:43.860008 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:58:43.860188 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:58:43.867895 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:58:43.868028 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:58:43.868230 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:58:43.878019 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36931
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:58:43.879328 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:58:43.880651 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:58:43.893997 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:43.893963 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:43.893955 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:58:43.894896 501 server_base.cc:1034] running on GCE node
I20260620 02:58:43.895480 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:58:43.896373 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:58:43.897629 501 hybrid_clock.cc:648] HybridClock initialized: now 1781924323897541 us; error 44 us; skew 500 ppm
I20260620 02:58:43.897958 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:58:43 dist-test-slave-f1j0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:58:43.901098 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:58:43.902555 501 webserver.cc:460] Webserver started at http://127.0.97.1:38295/ using document root <none> and password file <none>
I20260620 02:58:43.903226 501 fs_manager.cc:359] Metadata directory not provided
I20260620 02:58:43.903360 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:58:43.903671 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:58:43.906597 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "709d7961b28d4affb65eda288c5a64f1"
format_stamp: "Formatted at 2026-06-20 02:58:43 on dist-test-slave-f1j0"
server_key: "afd760ae980e0abf775055da45afc08c"
server_key_iv: "6a9e71a641088af980fa158ddf355514"
server_key_version: "encryptionkey@0"
I20260620 02:58:43.907362 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "709d7961b28d4affb65eda288c5a64f1"
format_stamp: "Formatted at 2026-06-20 02:58:43 on dist-test-slave-f1j0"
server_key: "afd760ae980e0abf775055da45afc08c"
server_key_iv: "6a9e71a641088af980fa158ddf355514"
server_key_version: "encryptionkey@0"
I20260620 02:58:43.912636 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:58:43.915916 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:43.917093 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 02:58:43.917264 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "709d7961b28d4affb65eda288c5a64f1"
format_stamp: "Formatted at 2026-06-20 02:58:43 on dist-test-slave-f1j0"
server_key: "afd760ae980e0abf775055da45afc08c"
server_key_iv: "6a9e71a641088af980fa158ddf355514"
server_key_version: "encryptionkey@0"
I20260620 02:58:43.917450 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:58:43.944522 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:58:43.948859 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:58:43.949149 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:58:43.950111 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:58:43.951373 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:58:43.951472 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:43.951612 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:58:43.951690 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:43.985524 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38469
I20260620 02:58:43.985556 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38469 every 8 connection(s)
I20260620 02:58:43.986824 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:58:43.995643 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 501
I20260620 02:58:43.995821 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:58:43.996430 388 external_mini_cluster.cc:1428] Setting key 85fd4a84b22420955d7a7ff06f85eaa6
Jun 20 02:58:43 dist-test-slave-f1j0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:44.007748 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:43.989237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56605 (local address 127.0.97.62:36931)
0620 02:58:43.989677 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:43.989681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:43.990899 (+ 1218us) server_negotiation.cc:407] Connection header received
0620 02:58:43.992670 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:43.992676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:43.992757 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:43.992880 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:43.994288 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:43.994978 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:43.996582 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:43.997146 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:43.997247 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:44.001760 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:44.001766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:44.001770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:44.001801 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:44.004554 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:44.005452 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:44.005456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:44.005458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:44.005623 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:44.006084 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:44.006087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:44.006089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:44.006343 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:44.006448 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:44.007331 (+ 883us) server_negotiation.cc:299] Negotiation successful
0620 02:58:44.007487 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":215,"threads_started":1}
I20260620 02:58:44.009371 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:43.989723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36931 (local address 127.0.97.1:56605)
0620 02:58:43.990741 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:43.990779 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:43.992412 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:43.993095 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:43.993102 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:43.993558 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:43.994130 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:43.994146 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:43.995126 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:43.995133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:43.996414 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:43.996429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:43.997362 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:43.997368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:43.997552 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:43.998335 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:43.998364 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:44.001514 (+ 3150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:44.004685 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:44.004697 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:44.004716 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:44.005335 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:44.005731 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:44.005739 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:44.005746 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:44.005978 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:44.006476 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:44.006494 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:44.006934 (+ 440us) client_negotiation.cc:769] Sending connection context
0620 02:58:44.007232 (+ 298us) client_negotiation.cc:240] Negotiation successful
0620 02:58:44.007817 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":616,"threads_started":1}
I20260620 02:58:44.011098 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36931
I20260620 02:58:44.011410 629 heartbeater.cc:464] Registering TS with master...
I20260620 02:58:44.012090 629 heartbeater.cc:511] Master 127.0.97.62:36931 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:58:44.014452 440 ts_manager.cc:194] Registered new tserver with Master: 709d7961b28d4affb65eda288c5a64f1 (127.0.97.1:38469)
I20260620 02:58:44.015956 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:58:44.036994 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36931
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:58:44.363498 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:58:44.363782 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:58:44.363878 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:58:44.371059 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:58:44.371162 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:58:44.371280 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:58:44.380941 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36931
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:58:44.382262 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:58:44.383539 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:58:44.399068 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:44.399291 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:44.399425 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:58:44.400794 635 server_base.cc:1034] running on GCE node
I20260620 02:58:44.401369 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:58:44.402361 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:58:44.403592 635 hybrid_clock.cc:648] HybridClock initialized: now 1781924324403558 us; error 61 us; skew 500 ppm
I20260620 02:58:44.403926 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:58:44 dist-test-slave-f1j0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:58:44.407114 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:58:44.408476 635 webserver.cc:460] Webserver started at http://127.0.97.2:43459/ using document root <none> and password file <none>
I20260620 02:58:44.409153 635 fs_manager.cc:359] Metadata directory not provided
I20260620 02:58:44.409293 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:58:44.409636 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:58:44.412508 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "40033236d8094f4d95b7fdedac2fc69d"
format_stamp: "Formatted at 2026-06-20 02:58:44 on dist-test-slave-f1j0"
server_key: "554478d34157cb47d1f897e1d4014869"
server_key_iv: "897ee1dcbb79e75f4da9db9f874ca5d2"
server_key_version: "encryptionkey@0"
I20260620 02:58:44.413267 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "40033236d8094f4d95b7fdedac2fc69d"
format_stamp: "Formatted at 2026-06-20 02:58:44 on dist-test-slave-f1j0"
server_key: "554478d34157cb47d1f897e1d4014869"
server_key_iv: "897ee1dcbb79e75f4da9db9f874ca5d2"
server_key_version: "encryptionkey@0"
I20260620 02:58:44.418592 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 02:58:44.422308 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:44.423556 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:58:44.423724 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "40033236d8094f4d95b7fdedac2fc69d"
format_stamp: "Formatted at 2026-06-20 02:58:44 on dist-test-slave-f1j0"
server_key: "554478d34157cb47d1f897e1d4014869"
server_key_iv: "897ee1dcbb79e75f4da9db9f874ca5d2"
server_key_version: "encryptionkey@0"
I20260620 02:58:44.423911 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:58:44.442981 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:58:44.446848 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:58:44.447140 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:58:44.448112 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:58:44.449823 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:58:44.449920 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:44.450058 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:58:44.450127 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:44.485252 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43249
I20260620 02:58:44.485296 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43249 every 8 connection(s)
I20260620 02:58:44.486538 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:58:44.492399 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 635
I20260620 02:58:44.492584 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:58:44.493151 388 external_mini_cluster.cc:1428] Setting key 7f6e52f96b7de16dfbd2bdcbfe2b6243
Jun 20 02:58:44 dist-test-slave-f1j0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:44.508225 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:44.489165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47481 (local address 127.0.97.62:36931)
0620 02:58:44.489355 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:44.489361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:44.490617 (+ 1256us) server_negotiation.cc:407] Connection header received
0620 02:58:44.492706 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:44.492713 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:44.492799 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:44.492889 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:44.494815 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:44.495503 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:44.497335 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:44.497959 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:44.498096 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:44.502601 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:44.502607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:44.502611 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:44.502642 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:44.505212 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:44.506003 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:44.506008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:44.506010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:44.506132 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:44.506547 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:44.506552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:44.506555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:44.506809 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:44.506968 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:44.507626 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 02:58:44.507850 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:58:44.509275 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:44.489762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36931 (local address 127.0.97.2:47481)
0620 02:58:44.490456 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:44.490494 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:44.492296 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:44.493168 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:44.493182 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:44.493908 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:44.494647 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:44.494665 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:44.495659 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:44.495664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:44.497159 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:44.497173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:44.498183 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:44.498190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:44.498430 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:44.499556 (+ 1126us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:44.499597 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:44.502372 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:44.505348 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:44.505357 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:44.505374 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:44.505874 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:44.506241 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:44.506247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:44.506251 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:44.506430 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:44.506941 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:44.506949 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:44.507282 (+ 333us) client_negotiation.cc:769] Sending connection context
0620 02:58:44.507541 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 02:58:44.508111 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":249,"threads_started":1}
I20260620 02:58:44.510833 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36931
I20260620 02:58:44.511154 763 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 02:58:44.511935 763 heartbeater.cc:511] Master 127.0.97.62:36931 requested a full tablet report, sending...
I20260620 02:58:44.513357 440 ts_manager.cc:194] Registered new tserver with Master: 40033236d8094f4d95b7fdedac2fc69d (127.0.97.2:43249)
I20260620 02:58:44.514213 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:58:44.532455 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36931
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:58:44.859278 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:58:44.859561 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:58:44.859658 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:58:44.867113 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:58:44.867223 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:58:44.867341 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:58:44.877094 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36931
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:58:44.878463 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:58:44.879813 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:58:44.892810 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:44.892853 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:44.892880 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:58:44.893623 768 server_base.cc:1034] running on GCE node
I20260620 02:58:44.894152 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:58:44.895010 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:58:44.896250 768 hybrid_clock.cc:648] HybridClock initialized: now 1781924324896209 us; error 37 us; skew 500 ppm
I20260620 02:58:44.896570 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:58:44 dist-test-slave-f1j0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:58:44.899622 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:58:44.900931 768 webserver.cc:460] Webserver started at http://127.0.97.3:35075/ using document root <none> and password file <none>
I20260620 02:58:44.901759 768 fs_manager.cc:359] Metadata directory not provided
I20260620 02:58:44.901911 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:58:44.902215 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:58:44.905017 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "a709811eafaa49fa91d4a1de97be7153"
format_stamp: "Formatted at 2026-06-20 02:58:44 on dist-test-slave-f1j0"
server_key: "bc63280f0a79c2b795b95f916c1d7321"
server_key_iv: "e260584c1635247a367d3c84049f08c7"
server_key_version: "encryptionkey@0"
I20260620 02:58:44.905845 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a709811eafaa49fa91d4a1de97be7153"
format_stamp: "Formatted at 2026-06-20 02:58:44 on dist-test-slave-f1j0"
server_key: "bc63280f0a79c2b795b95f916c1d7321"
server_key_iv: "e260584c1635247a367d3c84049f08c7"
server_key_version: "encryptionkey@0"
I20260620 02:58:44.911175 768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:58:44.914390 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:44.915643 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:58:44.915805 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "a709811eafaa49fa91d4a1de97be7153"
format_stamp: "Formatted at 2026-06-20 02:58:44 on dist-test-slave-f1j0"
server_key: "bc63280f0a79c2b795b95f916c1d7321"
server_key_iv: "e260584c1635247a367d3c84049f08c7"
server_key_version: "encryptionkey@0"
I20260620 02:58:44.915994 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:58:44.960353 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:58:44.964746 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:58:44.965051 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:58:44.966051 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:58:44.967335 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:58:44.967432 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:44.967583 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:58:44.967654 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:45.002691 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37529
I20260620 02:58:45.002733 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37529 every 8 connection(s)
I20260620 02:58:45.003964 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:58:45.011381 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 768
I20260620 02:58:45.011543 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:58:45.012075 388 external_mini_cluster.cc:1428] Setting key 964902252053e89dbf9375bb4637590b
Jun 20 02:58:45 dist-test-slave-f1j0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:45.018919 629 heartbeater.cc:503] Master 127.0.97.62:36931 was elected leader, sending a full tablet report...
I20260620 02:58:45.024484 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:45.006526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46663 (local address 127.0.97.62:36931)
0620 02:58:45.006699 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:45.006703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:45.007769 (+ 1066us) server_negotiation.cc:407] Connection header received
0620 02:58:45.010017 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:45.010022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:45.010113 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:45.010222 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:45.011993 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:45.012800 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:45.014570 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:45.015212 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:45.015368 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:45.018783 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:45.018788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:45.018792 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:45.018822 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:45.022015 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:45.022698 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:45.022702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:45.022704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:45.022819 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:45.023168 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:45.023171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:45.023173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:45.023405 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:45.023513 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:45.024109 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 02:58:45.024254 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:58:45.025419 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:45.006965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36931 (local address 127.0.97.3:46663)
0620 02:58:45.007608 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:45.007645 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:45.009753 (+ 2108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:45.010447 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:45.010455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:45.010995 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:45.011806 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:45.011834 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:45.012951 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:45.012957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:45.014431 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:45.014442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:45.015440 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:45.015445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:45.015646 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:45.016408 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:45.016430 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:45.018577 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:45.022189 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:45.022195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:45.022208 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:45.022596 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:45.022904 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:45.022908 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:45.022910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:45.023077 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:45.023507 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:45.023512 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:45.023799 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 02:58:45.024001 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 02:58:45.024369 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":248,"threads_started":1}
I20260620 02:58:45.026829 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36931
I20260620 02:58:45.027163 896 heartbeater.cc:464] Registering TS with master...
I20260620 02:58:45.028030 896 heartbeater.cc:511] Master 127.0.97.62:36931 requested a full tablet report, sending...
I20260620 02:58:45.029294 440 ts_manager.cc:194] Registered new tserver with Master: a709811eafaa49fa91d4a1de97be7153 (127.0.97.3:37529)
I20260620 02:58:45.030109 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46663
I20260620 02:58:45.040050 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:58:45 dist-test-slave-f1j0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924325, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:58:45 dist-test-slave-f1j0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924325, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:58:45.083794 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:45.068565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39800 (local address 127.0.97.1:38469)
0620 02:58:45.069185 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:45.069196 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:45.069303 (+ 107us) server_negotiation.cc:407] Connection header received
0620 02:58:45.069531 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:45.069539 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:45.069836 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:45.070014 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:45.071377 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:45.072407 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:45.073706 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:45.074261 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:45.074402 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:45.078156 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:45.078168 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:45.078182 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:45.078212 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:45.081108 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:45.081788 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:45.081795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:45.081797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:45.081908 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:45.082307 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:45.082312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:45.082315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:45.082628 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:45.082925 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:45.083212 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 02:58:45.083458 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":276,"threads_started":1}
W20260620 02:58:45.085341 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39800
Jun 20 02:58:45 dist-test-slave-f1j0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:58:45 dist-test-slave-f1j0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:58:45.128232 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:45.114204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39804 (local address 127.0.97.1:38469)
0620 02:58:45.114440 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:45.114447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:45.114637 (+ 190us) server_negotiation.cc:407] Connection header received
0620 02:58:45.114811 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:45.114816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:45.114923 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:45.115025 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:45.116333 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:45.117041 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:45.118282 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:45.118714 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:45.118862 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:45.123264 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:45.123270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:45.123273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:45.123308 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:45.125756 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:45.126356 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:45.126360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:45.126362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:45.126476 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:45.126919 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:45.126922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:45.126924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:45.127170 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:45.127342 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:45.127632 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 02:58:45.127843 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 02:58:45.129786 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 501
I20260620 02:58:45.142303 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:45.130709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39814 (local address 127.0.97.1:38469)
0620 02:58:45.130859 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:45.130863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:45.131172 (+ 309us) server_negotiation.cc:407] Connection header received
0620 02:58:45.131285 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:45.131288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:45.131338 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:45.131415 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:45.132940 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:45.133708 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:45.134891 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:45.135293 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:45.135380 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:45.137685 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:45.137690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:45.137693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:45.137721 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:45.140197 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:45.140756 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:45.140759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:45.140761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:45.140871 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:45.141206 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:45.141209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:45.141211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:45.141432 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:45.141668 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:45.141895 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 02:58:45.142040 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 02:58:45.143059 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:58:45.224963 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 501
I20260620 02:58:45.241189 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 635
Jun 20 02:58:45 dist-test-slave-f1j0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:58:45.256302 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:45.242114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43520 (local address 127.0.97.2:43249)
0620 02:58:45.242608 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:45.242614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:45.242639 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:58:45.242707 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:45.242713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:45.242919 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:45.243039 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:45.244210 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:45.245214 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:45.246570 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:45.247089 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:45.247181 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:45.251053 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:45.251063 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:45.251078 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:45.251109 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:45.253957 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:45.254504 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:45.254510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:45.254513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:45.254618 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:45.254925 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:45.254928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:45.254930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:45.255230 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:45.255449 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:45.255833 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 02:58:45.256015 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":254,"threads_started":1}
I20260620 02:58:45.257555 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:58:45.339571 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 635
I20260620 02:58:45.361413 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 768
Jun 20 02:58:45 dist-test-slave-f1j0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:58:45.376938 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:45.362453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49648 (local address 127.0.97.3:37529)
0620 02:58:45.363076 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:45.363083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:45.363112 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:58:45.363167 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:45.363173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:45.363389 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:45.363507 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:45.364674 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:45.365824 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:45.367051 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:45.367577 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:45.367678 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:45.371517 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:45.371527 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:45.371541 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:45.371572 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:45.374448 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:45.374972 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:45.374978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:45.374980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:45.375082 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:45.375397 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:45.375403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:45.375406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:45.375808 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:45.376106 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:45.376295 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 02:58:45.376551 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":333,"threads_started":1}
I20260620 02:58:45.378301 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:58:45.462241 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 768
I20260620 02:58:45.477982 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 408
I20260620 02:58:45.479121 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:58:45.579555 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 408
2026-06-20T02:58:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5182 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:58:45 dist-test-slave-f1j0 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:58:45 dist-test-slave-f1j0 krb5kdc[917](info): set up 2 sockets
Jun 20 02:58:45 dist-test-slave-f1j0 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 02:58:46.665757 388 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 02:58:46 dist-test-slave-f1j0 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:58:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:58:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:58:46.841343 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38681
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:58:47.151397 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:58:47.151767 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:58:47.151942 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:58:47.159080 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:58:47.159241 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:58:47.159354 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:58:47.159457 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:58:47.159546 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:58:47.169158 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:58:47.170478 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:58:47.171772 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:58:47.183250 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:47.183331 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:47.183276 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:58:47.184494 930 server_base.cc:1034] running on GCE node
I20260620 02:58:47.185313 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:58:47.186734 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:58:47.187937 930 hybrid_clock.cc:648] HybridClock initialized: now 1781924327187883 us; error 52 us; skew 500 ppm
I20260620 02:58:47.188298 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:58:47 dist-test-slave-f1j0 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:58:47.191380 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:58:47.192823 930 webserver.cc:460] Webserver started at http://127.0.97.62:43211/ using document root <none> and password file <none>
I20260620 02:58:47.193478 930 fs_manager.cc:359] Metadata directory not provided
I20260620 02:58:47.193627 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:58:47.193938 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:58:47.196723 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "7eec05e7f4b34ca983cf402ad53fca68"
format_stamp: "Formatted at 2026-06-20 02:58:47 on dist-test-slave-f1j0"
server_key: "f13317cbb63b631e1243b493b450c2ab"
server_key_iv: "2c456510f979ead20ec3a2f7d3c38475"
server_key_version: "encryptionkey@0"
I20260620 02:58:47.197438 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "7eec05e7f4b34ca983cf402ad53fca68"
format_stamp: "Formatted at 2026-06-20 02:58:47 on dist-test-slave-f1j0"
server_key: "f13317cbb63b631e1243b493b450c2ab"
server_key_iv: "2c456510f979ead20ec3a2f7d3c38475"
server_key_version: "encryptionkey@0"
I20260620 02:58:47.202369 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 02:58:47.205754 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:47.207157 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:58:47.207315 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "7eec05e7f4b34ca983cf402ad53fca68"
format_stamp: "Formatted at 2026-06-20 02:58:47 on dist-test-slave-f1j0"
server_key: "f13317cbb63b631e1243b493b450c2ab"
server_key_iv: "2c456510f979ead20ec3a2f7d3c38475"
server_key_version: "encryptionkey@0"
I20260620 02:58:47.207481 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:58:47.237007 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:58:47.241134 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:58:47.241443 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:58:47.259521 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38681
I20260620 02:58:47.259563 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38681 every 8 connection(s)
I20260620 02:58:47.260864 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:58:47.264621 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:47.267198 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 930
I20260620 02:58:47.267405 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 02:58:47.268229 388 external_mini_cluster.cc:1428] Setting key db193de19c11493438699eb99e7ae881
I20260620 02:58:47.277727 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68: Bootstrap starting.
I20260620 02:58:47.281533 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68: Neither blocks nor log segments found. Creating new log.
Jun 20 02:58:47 dist-test-slave-f1j0 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:47.282685 997 log.cc:824] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68: Log is configured to *not* fsync() on all Append() calls
I20260620 02:58:47.286026 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68: No bootstrap required, opened a new log
I20260620 02:58:47.291970 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:47.270276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59278 (local address 127.0.97.62:38681)
0620 02:58:47.271069 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:47.271080 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:47.271119 (+ 39us) server_negotiation.cc:407] Connection header received
0620 02:58:47.272710 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:47.272730 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:47.273094 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:47.273446 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:47.275211 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:47.276584 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:47.278314 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:47.278931 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:47.279051 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:47.283669 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:47.283676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:47.283692 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:47.283727 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:47.287640 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:47.288248 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:47.288261 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:47.288269 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:47.288478 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:47.288798 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:47.288803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:47.288806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:47.289491 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:47.289797 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:47.290210 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 02:58:47.290644 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":375,"threads_started":1}
I20260620 02:58:47.295434 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eec05e7f4b34ca983cf402ad53fca68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38681 } }
I20260620 02:58:47.295768 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:47.295883 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eec05e7f4b34ca983cf402ad53fca68, State: Initialized, Role: FOLLOWER
I20260620 02:58:47.296401 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eec05e7f4b34ca983cf402ad53fca68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38681 } }
I20260620 02:58:47.296634 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:58:47.296759 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:58:47.296923 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:47.299326 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eec05e7f4b34ca983cf402ad53fca68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38681 } }
I20260620 02:58:47.299799 997 leader_election.cc:302] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eec05e7f4b34ca983cf402ad53fca68; no voters:
I20260620 02:58:47.300329 997 leader_election.cc:288] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:58:47.300463 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:58:47.301153 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68 [term 1 LEADER]: Becoming Leader. State: Replica: 7eec05e7f4b34ca983cf402ad53fca68, State: Running, Role: LEADER
I20260620 02:58:47.301684 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eec05e7f4b34ca983cf402ad53fca68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38681 } }
I20260620 02:58:47.302335 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:58:47.304728 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eec05e7f4b34ca983cf402ad53fca68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eec05e7f4b34ca983cf402ad53fca68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38681 } } }
I20260620 02:58:47.305084 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68 [sys.catalog]: This master's current role is: LEADER
I20260620 02:58:47.304824 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eec05e7f4b34ca983cf402ad53fca68. Latest consensus state: current_term: 1 leader_uuid: "7eec05e7f4b34ca983cf402ad53fca68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eec05e7f4b34ca983cf402ad53fca68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38681 } } }
I20260620 02:58:47.305212 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68 [sys.catalog]: This master's current role is: LEADER
I20260620 02:58:47.306125 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:58:47.312711 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:58:47.323267 1008 catalog_manager.cc:1347] Generated new cluster ID: 5ac9eed8b07542dfadc8066e0f8d3e0d
I20260620 02:58:47.323408 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:58:47.346975 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:58:47.348344 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:58:47.354328 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7eec05e7f4b34ca983cf402ad53fca68: Generated new TSK 0
I20260620 02:58:47.355332 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:58:47.398582 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38681
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:58:47.729956 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:58:47.730242 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:58:47.730415 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:58:47.737630 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:58:47.737778 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:58:47.737975 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:58:47.748054 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38681
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:58:47.749437 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:58:47.750814 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:58:47.764348 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:47.764274 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:47.764801 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:58:47.764935 1023 server_base.cc:1034] running on GCE node
I20260620 02:58:47.765542 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:58:47.766479 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:58:47.767674 1023 hybrid_clock.cc:648] HybridClock initialized: now 1781924327767618 us; error 61 us; skew 500 ppm
I20260620 02:58:47.768005 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:58:47 dist-test-slave-f1j0 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:58:47.771077 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:58:47.772534 1023 webserver.cc:460] Webserver started at http://127.0.97.1:45919/ using document root <none> and password file <none>
I20260620 02:58:47.773237 1023 fs_manager.cc:359] Metadata directory not provided
I20260620 02:58:47.773380 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:58:47.773756 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:58:47.776618 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "b184657dd82540f0b211ab8716d8a791"
format_stamp: "Formatted at 2026-06-20 02:58:47 on dist-test-slave-f1j0"
server_key: "e81282280c3c9bf94b1c8ced4e03119a"
server_key_iv: "4be64bdb03d748e760940a1b461ee5a6"
server_key_version: "encryptionkey@0"
I20260620 02:58:47.777395 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b184657dd82540f0b211ab8716d8a791"
format_stamp: "Formatted at 2026-06-20 02:58:47 on dist-test-slave-f1j0"
server_key: "e81282280c3c9bf94b1c8ced4e03119a"
server_key_iv: "4be64bdb03d748e760940a1b461ee5a6"
server_key_version: "encryptionkey@0"
I20260620 02:58:47.782621 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:58:47.786010 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:47.787271 1023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 02:58:47.787448 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "b184657dd82540f0b211ab8716d8a791"
format_stamp: "Formatted at 2026-06-20 02:58:47 on dist-test-slave-f1j0"
server_key: "e81282280c3c9bf94b1c8ced4e03119a"
server_key_iv: "4be64bdb03d748e760940a1b461ee5a6"
server_key_version: "encryptionkey@0"
I20260620 02:58:47.787652 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:58:47.804828 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:58:47.809013 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:58:47.809314 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:58:47.810249 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:58:47.811547 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:58:47.811620 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:47.811784 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:58:47.811851 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:47.843030 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34495
I20260620 02:58:47.843058 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34495 every 8 connection(s)
I20260620 02:58:47.844322 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:58:47.847215 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 1023
I20260620 02:58:47.847431 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:58:47.848052 388 external_mini_cluster.cc:1428] Setting key c238a8022616b1d36136a6c764293bb0
Jun 20 02:58:47 dist-test-slave-f1j0 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:58:47.864975 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:47.846887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54403 (local address 127.0.97.62:38681)
0620 02:58:47.847372 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:47.847378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:47.848810 (+ 1432us) server_negotiation.cc:407] Connection header received
0620 02:58:47.850520 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:47.850528 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:47.850619 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:47.850744 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:47.852293 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:47.853509 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:47.855287 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:47.855683 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:47.855828 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:47.859253 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:47.859258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:47.859261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:47.859292 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:47.862045 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:47.862776 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:47.862788 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:47.862791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:47.862908 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:47.863388 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:47.863394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:47.863397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:47.863727 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:47.863841 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:47.864565 (+ 724us) server_negotiation.cc:299] Negotiation successful
0620 02:58:47.864729 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":243,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:58:47.866120 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:47.847483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38681 (local address 127.0.97.1:54403)
0620 02:58:47.848391 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:47.848446 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:47.850259 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:47.851001 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:47.851012 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:47.851562 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:47.852123 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:47.852141 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:47.853715 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:47.853719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:47.855131 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:47.855143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:47.855920 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:47.855926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:47.856154 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:47.856847 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:47.856870 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:47.859008 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:47.862190 (+ 3182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:47.862199 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:47.862217 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:47.862643 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:47.863046 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:47.863051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:47.863061 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:47.863238 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:47.863841 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:47.863847 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:47.864179 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 02:58:47.864475 (+ 296us) client_negotiation.cc:240] Negotiation successful
0620 02:58:47.864876 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":354,"threads_started":1}
I20260620 02:58:47.867751 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38681
I20260620 02:58:47.868099 1151 heartbeater.cc:464] Registering TS with master...
I20260620 02:58:47.868969 1151 heartbeater.cc:511] Master 127.0.97.62:38681 requested a full tablet report, sending...
I20260620 02:58:47.871477 962 ts_manager.cc:194] Registered new tserver with Master: b184657dd82540f0b211ab8716d8a791 (127.0.97.1:34495)
I20260620 02:58:47.873246 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:58:47.889132 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38681
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:58:48.204869 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:58:48.205183 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:58:48.205358 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:58:48.212807 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:58:48.212960 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:58:48.213143 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:58:48.222827 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38681
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:58:48.224107 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:58:48.225414 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:58:48.238384 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:48.238444 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:48.238488 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:58:48.239326 1157 server_base.cc:1034] running on GCE node
I20260620 02:58:48.239966 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:58:48.240927 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:58:48.242192 1157 hybrid_clock.cc:648] HybridClock initialized: now 1781924328242140 us; error 44 us; skew 500 ppm
I20260620 02:58:48.242556 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:58:48 dist-test-slave-f1j0 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:58:48.245882 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:58:48.247316 1157 webserver.cc:460] Webserver started at http://127.0.97.2:43115/ using document root <none> and password file <none>
I20260620 02:58:48.248036 1157 fs_manager.cc:359] Metadata directory not provided
I20260620 02:58:48.248168 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:58:48.248503 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:58:48.251485 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "45fcaa7f8b0a41468dd77136041d11f7"
format_stamp: "Formatted at 2026-06-20 02:58:48 on dist-test-slave-f1j0"
server_key: "95627fbbeeda7545038d0e10cf5da01a"
server_key_iv: "d1e24875a6b39b964ea0e089c3d5c7bb"
server_key_version: "encryptionkey@0"
I20260620 02:58:48.252281 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "45fcaa7f8b0a41468dd77136041d11f7"
format_stamp: "Formatted at 2026-06-20 02:58:48 on dist-test-slave-f1j0"
server_key: "95627fbbeeda7545038d0e10cf5da01a"
server_key_iv: "d1e24875a6b39b964ea0e089c3d5c7bb"
server_key_version: "encryptionkey@0"
I20260620 02:58:48.257440 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:58:48.260980 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:48.262454 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:58:48.262611 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "45fcaa7f8b0a41468dd77136041d11f7"
format_stamp: "Formatted at 2026-06-20 02:58:48 on dist-test-slave-f1j0"
server_key: "95627fbbeeda7545038d0e10cf5da01a"
server_key_iv: "d1e24875a6b39b964ea0e089c3d5c7bb"
server_key_version: "encryptionkey@0"
I20260620 02:58:48.262792 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:58:48.293295 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:58:48.297855 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:58:48.298135 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:58:48.299108 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:58:48.300405 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:58:48.300475 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:48.300599 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:58:48.300658 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:48.333709 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33251
I20260620 02:58:48.333835 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33251 every 8 connection(s)
I20260620 02:58:48.334990 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:58:48.335691 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 1157
I20260620 02:58:48.335876 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:58:48.336439 388 external_mini_cluster.cc:1428] Setting key bf485591c4f05f6f29a7243ae5778a30
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Jun 20 02:58:48 dist-test-slave-f1j0 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 02:58:48.359397 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:48.338233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49269 (local address 127.0.97.62:38681)
0620 02:58:48.338411 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:48.338415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:48.339679 (+ 1264us) server_negotiation.cc:407] Connection header received
0620 02:58:48.341467 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:48.341476 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:48.341598 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:48.341710 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:48.343126 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.344025 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.345554 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.345990 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.346123 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:48.353271 (+ 7148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:48.353277 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:48.353280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:48.353317 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:48.356251 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:48.357032 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:48.357038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:48.357042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:48.357184 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:48.357673 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:48.357678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:48.357687 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:48.357934 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:48.358076 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:48.358833 (+ 757us) server_negotiation.cc:299] Negotiation successful
0620 02:58:48.359057 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:58:48.360500 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:48.338715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38681 (local address 127.0.97.2:49269)
0620 02:58:48.339510 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:48.339555 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:48.341210 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:48.341905 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:48.341913 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:48.342403 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:48.342949 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:48.342966 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.344162 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.344171 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:48.345397 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:48.345409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.346233 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.346240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:48.346468 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:48.350740 (+ 4272us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:48.350768 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:48.352950 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:48.356427 (+ 3477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:48.356437 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:48.356456 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:48.356899 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:48.357309 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:48.357314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:48.357317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:48.357519 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:48.358067 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:48.358076 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:48.358449 (+ 373us) client_negotiation.cc:769] Sending connection context
0620 02:58:48.358736 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 02:58:48.359313 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":360,"threads_started":1}
I20260620 02:58:48.362071 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38681
I20260620 02:58:48.362422 1285 heartbeater.cc:464] Registering TS with master...
I20260620 02:58:48.363306 1285 heartbeater.cc:511] Master 127.0.97.62:38681 requested a full tablet report, sending...
I20260620 02:58:48.364967 962 ts_manager.cc:194] Registered new tserver with Master: 45fcaa7f8b0a41468dd77136041d11f7 (127.0.97.2:33251)
I20260620 02:58:48.366278 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:58:48.377496 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38681
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:58:48.695286 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:58:48.695575 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:58:48.695740 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:58:48.703095 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:58:48.703231 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:58:48.703429 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:58:48.713560 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38681
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:58:48.714913 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:58:48.716264 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:58:48.729693 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:48.729674 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:48.729771 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:58:48.730435 1290 server_base.cc:1034] running on GCE node
I20260620 02:58:48.731072 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:58:48.732030 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:58:48.733258 1290 hybrid_clock.cc:648] HybridClock initialized: now 1781924328733194 us; error 44 us; skew 500 ppm
I20260620 02:58:48.733696 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:58:48 dist-test-slave-f1j0 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:58:48.737023 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:58:48.738703 1290 webserver.cc:460] Webserver started at http://127.0.97.3:41009/ using document root <none> and password file <none>
I20260620 02:58:48.739485 1290 fs_manager.cc:359] Metadata directory not provided
I20260620 02:58:48.739626 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:58:48.739975 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:58:48.743424 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "12747e7f712843f4ba8d8a9ff24c9ee2"
format_stamp: "Formatted at 2026-06-20 02:58:48 on dist-test-slave-f1j0"
server_key: "2f4b6824c344e89477672ef5fa2c214a"
server_key_iv: "51614f438b0ba334892b42134cc4a2aa"
server_key_version: "encryptionkey@0"
I20260620 02:58:48.744274 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "12747e7f712843f4ba8d8a9ff24c9ee2"
format_stamp: "Formatted at 2026-06-20 02:58:48 on dist-test-slave-f1j0"
server_key: "2f4b6824c344e89477672ef5fa2c214a"
server_key_iv: "51614f438b0ba334892b42134cc4a2aa"
server_key_version: "encryptionkey@0"
I20260620 02:58:48.750176 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 02:58:48.753901 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:48.755272 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:58:48.755467 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "12747e7f712843f4ba8d8a9ff24c9ee2"
format_stamp: "Formatted at 2026-06-20 02:58:48 on dist-test-slave-f1j0"
server_key: "2f4b6824c344e89477672ef5fa2c214a"
server_key_iv: "51614f438b0ba334892b42134cc4a2aa"
server_key_version: "encryptionkey@0"
I20260620 02:58:48.755695 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:58:48.791687 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:58:48.796020 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:58:48.796344 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:58:48.797343 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:58:48.798799 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:58:48.798928 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:48.799093 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:58:48.799178 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:48.834017 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45249
I20260620 02:58:48.834059 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45249 every 8 connection(s)
I20260620 02:58:48.835386 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:58:48.845147 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 1290
I20260620 02:58:48.845321 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:58:48.845947 388 external_mini_cluster.cc:1428] Setting key 0561420ee96ec2be5d4d04dfd0060b60
Jun 20 02:58:48 dist-test-slave-f1j0 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:48.855968 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:48.838415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59521 (local address 127.0.97.62:38681)
0620 02:58:48.838577 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:48.838582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:48.839798 (+ 1216us) server_negotiation.cc:407] Connection header received
0620 02:58:48.841504 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:48.841509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:48.841637 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:48.841742 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:48.843161 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.843914 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.845605 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.846028 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.846149 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:48.850393 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:48.850398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:48.850401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:48.850433 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:48.853349 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:48.854075 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:48.854079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:48.854081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:48.854208 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:48.854573 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:48.854576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:48.854578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:48.854829 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:48.854930 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:48.855576 (+ 646us) server_negotiation.cc:299] Negotiation successful
0620 02:58:48.855740 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 02:58:48.857029 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:48.838926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38681 (local address 127.0.97.3:59521)
0620 02:58:48.839617 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:48.839661 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:48.841239 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:48.841941 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:48.841949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:48.842454 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:48.842993 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:48.843009 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.844063 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.844070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:48.845395 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:48.845411 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.846246 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.846253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:48.846558 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:48.847510 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:48.847543 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:48.850199 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:48.853472 (+ 3273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:48.853479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:48.853491 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:48.853976 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:48.854301 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:48.854305 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:48.854307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:48.854485 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:48.854929 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:48.854935 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:48.855256 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 02:58:48.855496 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 02:58:48.855904 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":295,"threads_started":1}
I20260620 02:58:48.858407 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38681
I20260620 02:58:48.858703 1418 heartbeater.cc:464] Registering TS with master...
I20260620 02:58:48.859474 1418 heartbeater.cc:511] Master 127.0.97.62:38681 requested a full tablet report, sending...
I20260620 02:58:48.861158 962 ts_manager.cc:194] Registered new tserver with Master: 12747e7f712843f4ba8d8a9ff24c9ee2 (127.0.97.3:45249)
I20260620 02:58:48.862144 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59521
I20260620 02:58:48.873919 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:58:48.877022 1151 heartbeater.cc:503] Master 127.0.97.62:38681 was elected leader, sending a full tablet report...
I20260620 02:58:48.892580 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:48.880496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.0.97.62:38681)
0620 02:58:48.880644 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:48.880648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:48.881141 (+ 493us) server_negotiation.cc:407] Connection header received
0620 02:58:48.881274 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:48.881277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:48.881335 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:48.881417 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:48.882848 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.883543 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.884725 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.885095 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.885180 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:48.887811 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:48.887816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:48.887819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:48.887850 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:48.890417 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:48.890945 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:48.890949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:48.890952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:48.891075 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:48.891386 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:48.891391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:48.891394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:48.891688 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:48.891922 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:48.892201 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 02:58:48.892371 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 02:58:48.898655 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:58:48.905462 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:58:48.933023 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:48.917695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34495 (local address 127.0.0.1:45022)
0620 02:58:48.919832 (+ 2137us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:48.919865 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:48.920034 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:48.925086 (+ 5052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:48.925093 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:48.925122 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:48.925454 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:48.925463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.928021 (+ 2558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.928028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:48.930494 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:48.930511 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.931957 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.931964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:48.932223 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:48.932269 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 02:58:48.932405 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 02:58:48.932594 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1922,"mutex_wait_us":1,"thread_start_us":8070,"threads_started":1}
I20260620 02:58:48.932936 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:48.917903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45022 (local address 127.0.97.1:34495)
0620 02:58:48.924589 (+ 6686us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:48.924595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:48.924618 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:58:48.924694 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:48.924699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:48.924880 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:48.925081 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:48.925625 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.927880 (+ 2255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.930662 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.931731 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.931887 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:48.932136 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:48.932445 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 02:58:48.932610 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6539,"thread_start_us":6397,"threads_started":1}
I20260620 02:58:48.934818 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:48.918318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45249 (local address 127.0.0.1:57244)
0620 02:58:48.926020 (+ 7702us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:48.926040 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:48.926164 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:48.927131 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:48.927139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:48.927163 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:48.928052 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:48.928065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.930009 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.930019 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:48.932027 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:48.932042 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.934085 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.934091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:48.934245 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:48.934277 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:58:48.934369 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 02:58:48.934487 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7582,"thread_start_us":1151,"threads_started":1}
I20260620 02:58:48.934944 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:48.918732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.0.97.3:45249)
0620 02:58:48.919577 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:48.919590 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:48.926303 (+ 6713us) server_negotiation.cc:407] Connection header received
0620 02:58:48.926429 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:48.926440 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:48.926685 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:48.926865 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:48.928254 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.929840 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.932200 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.933621 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.933726 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:48.933922 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:48.934436 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 02:58:48.934593 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":419,"threads_started":1}
I20260620 02:58:48.937126 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:48.926190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33251 (local address 127.0.0.1:41224)
0620 02:58:48.926789 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:48.926811 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:48.926944 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:48.928659 (+ 1715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:48.928663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:48.928689 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:48.929054 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:48.929062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.931043 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.931049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:48.933188 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:48.933201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.936395 (+ 3194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.936401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:48.936559 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:48.936587 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 02:58:48.936682 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 02:58:48.936808 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":51,"thread_start_us":261,"threads_started":1}
I20260620 02:58:48.937920 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:48.926479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.0.97.2:33251)
0620 02:58:48.928207 (+ 1728us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:48.928218 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:48.928241 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:58:48.928328 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:48.928333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:48.928489 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:48.928677 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:48.929222 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.930906 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.933358 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:48.935984 (+ 2626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:48.936099 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:48.936269 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:48.937423 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0620 02:58:48.937610 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1560,"thread_start_us":961,"threads_started":1}
I20260620 02:58:48.937837 1086 tablet_service.cc:1467] Processing CreateTablet for tablet aaf47d252ef54fa88c6c804c22bb3a01 (DEFAULT_TABLE table=test-table [id=8b849e2de9b04040a6e0454dcce29dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:58:48.938036 1353 tablet_service.cc:1467] Processing CreateTablet for tablet aaf47d252ef54fa88c6c804c22bb3a01 (DEFAULT_TABLE table=test-table [id=8b849e2de9b04040a6e0454dcce29dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:58:48.939796 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaf47d252ef54fa88c6c804c22bb3a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:48.939790 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaf47d252ef54fa88c6c804c22bb3a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:48.941596 1220 tablet_service.cc:1467] Processing CreateTablet for tablet aaf47d252ef54fa88c6c804c22bb3a01 (DEFAULT_TABLE table=test-table [id=8b849e2de9b04040a6e0454dcce29dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:58:48.942848 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaf47d252ef54fa88c6c804c22bb3a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:48.950873 1436 tablet_bootstrap.cc:492] T aaf47d252ef54fa88c6c804c22bb3a01 P 12747e7f712843f4ba8d8a9ff24c9ee2: Bootstrap starting.
I20260620 02:58:48.954152 1436 tablet_bootstrap.cc:654] T aaf47d252ef54fa88c6c804c22bb3a01 P 12747e7f712843f4ba8d8a9ff24c9ee2: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:48.954169 1437 tablet_bootstrap.cc:492] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791: Bootstrap starting.
I20260620 02:58:48.956971 1436 log.cc:824] T aaf47d252ef54fa88c6c804c22bb3a01 P 12747e7f712843f4ba8d8a9ff24c9ee2: Log is configured to *not* fsync() on all Append() calls
I20260620 02:58:48.958241 1437 tablet_bootstrap.cc:654] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:48.959889 1437 log.cc:824] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791: Log is configured to *not* fsync() on all Append() calls
I20260620 02:58:48.962293 1438 tablet_bootstrap.cc:492] T aaf47d252ef54fa88c6c804c22bb3a01 P 45fcaa7f8b0a41468dd77136041d11f7: Bootstrap starting.
I20260620 02:58:48.965785 1438 tablet_bootstrap.cc:654] T aaf47d252ef54fa88c6c804c22bb3a01 P 45fcaa7f8b0a41468dd77136041d11f7: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:48.965849 1436 tablet_bootstrap.cc:492] T aaf47d252ef54fa88c6c804c22bb3a01 P 12747e7f712843f4ba8d8a9ff24c9ee2: No bootstrap required, opened a new log
I20260620 02:58:48.965849 1437 tablet_bootstrap.cc:492] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791: No bootstrap required, opened a new log
I20260620 02:58:48.966121 1437 ts_tablet_manager.cc:1397] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20260620 02:58:48.966125 1436 ts_tablet_manager.cc:1397] T aaf47d252ef54fa88c6c804c22bb3a01 P 12747e7f712843f4ba8d8a9ff24c9ee2: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260620 02:58:48.967358 1438 log.cc:824] T aaf47d252ef54fa88c6c804c22bb3a01 P 45fcaa7f8b0a41468dd77136041d11f7: Log is configured to *not* fsync() on all Append() calls
I20260620 02:58:48.969885 1438 tablet_bootstrap.cc:492] T aaf47d252ef54fa88c6c804c22bb3a01 P 45fcaa7f8b0a41468dd77136041d11f7: No bootstrap required, opened a new log
I20260620 02:58:48.970129 1438 ts_tablet_manager.cc:1397] T aaf47d252ef54fa88c6c804c22bb3a01 P 45fcaa7f8b0a41468dd77136041d11f7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260620 02:58:48.976886 1437 raft_consensus.cc:348] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b184657dd82540f0b211ab8716d8a791" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34495 } } peers { permanent_uuid: "45fcaa7f8b0a41468dd77136041d11f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33251 } } peers { permanent_uuid: "12747e7f712843f4ba8d8a9ff24c9ee2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45249 } }
I20260620 02:58:48.976886 1436 raft_consensus.cc:348] T aaf47d252ef54fa88c6c804c22bb3a01 P 12747e7f712843f4ba8d8a9ff24c9ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b184657dd82540f0b211ab8716d8a791" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34495 } } peers { permanent_uuid: "45fcaa7f8b0a41468dd77136041d11f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33251 } } peers { permanent_uuid: "12747e7f712843f4ba8d8a9ff24c9ee2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45249 } }
I20260620 02:58:48.977336 1436 raft_consensus.cc:374] T aaf47d252ef54fa88c6c804c22bb3a01 P 12747e7f712843f4ba8d8a9ff24c9ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:48.977336 1437 raft_consensus.cc:374] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:48.977437 1436 raft_consensus.cc:729] T aaf47d252ef54fa88c6c804c22bb3a01 P 12747e7f712843f4ba8d8a9ff24c9ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12747e7f712843f4ba8d8a9ff24c9ee2, State: Initialized, Role: FOLLOWER
I20260620 02:58:48.977437 1437 raft_consensus.cc:729] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b184657dd82540f0b211ab8716d8a791, State: Initialized, Role: FOLLOWER
I20260620 02:58:48.977972 1438 raft_consensus.cc:348] T aaf47d252ef54fa88c6c804c22bb3a01 P 45fcaa7f8b0a41468dd77136041d11f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b184657dd82540f0b211ab8716d8a791" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34495 } } peers { permanent_uuid: "45fcaa7f8b0a41468dd77136041d11f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33251 } } peers { permanent_uuid: "12747e7f712843f4ba8d8a9ff24c9ee2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45249 } }
I20260620 02:58:48.978096 1436 consensus_queue.cc:260] T aaf47d252ef54fa88c6c804c22bb3a01 P 12747e7f712843f4ba8d8a9ff24c9ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b184657dd82540f0b211ab8716d8a791" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34495 } } peers { permanent_uuid: "45fcaa7f8b0a41468dd77136041d11f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33251 } } peers { permanent_uuid: "12747e7f712843f4ba8d8a9ff24c9ee2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45249 } }
I20260620 02:58:48.978084 1437 consensus_queue.cc:260] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b184657dd82540f0b211ab8716d8a791" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34495 } } peers { permanent_uuid: "45fcaa7f8b0a41468dd77136041d11f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33251 } } peers { permanent_uuid: "12747e7f712843f4ba8d8a9ff24c9ee2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45249 } }
I20260620 02:58:48.978294 1438 raft_consensus.cc:374] T aaf47d252ef54fa88c6c804c22bb3a01 P 45fcaa7f8b0a41468dd77136041d11f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:48.978412 1438 raft_consensus.cc:729] T aaf47d252ef54fa88c6c804c22bb3a01 P 45fcaa7f8b0a41468dd77136041d11f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45fcaa7f8b0a41468dd77136041d11f7, State: Initialized, Role: FOLLOWER
I20260620 02:58:48.978953 1438 consensus_queue.cc:260] T aaf47d252ef54fa88c6c804c22bb3a01 P 45fcaa7f8b0a41468dd77136041d11f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b184657dd82540f0b211ab8716d8a791" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34495 } } peers { permanent_uuid: "45fcaa7f8b0a41468dd77136041d11f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33251 } } peers { permanent_uuid: "12747e7f712843f4ba8d8a9ff24c9ee2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45249 } }
I20260620 02:58:48.979966 1285 heartbeater.cc:503] Master 127.0.97.62:38681 was elected leader, sending a full tablet report...
I20260620 02:58:48.982240 1438 ts_tablet_manager.cc:1428] T aaf47d252ef54fa88c6c804c22bb3a01 P 45fcaa7f8b0a41468dd77136041d11f7: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260620 02:58:48.985102 1418 heartbeater.cc:503] Master 127.0.97.62:38681 was elected leader, sending a full tablet report...
I20260620 02:58:48.985384 1437 ts_tablet_manager.cc:1428] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791: Time spent starting tablet: real 0.019s user 0.017s sys 0.001s
I20260620 02:58:48.985656 1436 ts_tablet_manager.cc:1428] T aaf47d252ef54fa88c6c804c22bb3a01 P 12747e7f712843f4ba8d8a9ff24c9ee2: Time spent starting tablet: real 0.019s user 0.015s sys 0.005s
I20260620 02:58:49.011510 1442 raft_consensus.cc:482] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:58:49.011827 1442 raft_consensus.cc:504] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b184657dd82540f0b211ab8716d8a791" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34495 } } peers { permanent_uuid: "45fcaa7f8b0a41468dd77136041d11f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33251 } } peers { permanent_uuid: "12747e7f712843f4ba8d8a9ff24c9ee2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45249 } }
I20260620 02:58:49.013468 1442 leader_election.cc:288] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45fcaa7f8b0a41468dd77136041d11f7 (127.0.97.2:33251), 12747e7f712843f4ba8d8a9ff24c9ee2 (127.0.97.3:45249)
I20260620 02:58:49.021677 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:49.013912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37925 (local address 127.0.97.2:33251)
0620 02:58:49.014113 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:49.014120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:49.014828 (+ 708us) server_negotiation.cc:407] Connection header received
0620 02:58:49.015021 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:49.015026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:49.015096 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:49.015204 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:49.015967 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.017055 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.019367 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.020134 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.020316 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:49.020440 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:49.021248 (+ 808us) server_negotiation.cc:299] Negotiation successful
0620 02:58:49.021384 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:58:49.021719 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:49.013920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36573 (local address 127.0.97.3:45249)
0620 02:58:49.014113 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:49.014119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:49.014828 (+ 709us) server_negotiation.cc:407] Connection header received
0620 02:58:49.015017 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:49.015022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:49.015096 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:49.015195 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:49.015967 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.017060 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.019383 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.020215 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.020365 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:49.020487 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:49.021338 (+ 851us) server_negotiation.cc:299] Negotiation successful
0620 02:58:49.021470 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:58:49.021795 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:49.013832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33251 (local address 127.0.97.1:37925)
0620 02:58:49.014719 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:49.014736 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:49.014903 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:49.015227 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:49.015232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:49.015278 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:49.015820 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:49.015828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.017193 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.017196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:49.019212 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:49.019226 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.020891 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.020898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:49.021030 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:49.021091 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 02:58:49.021203 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 02:58:49.021357 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":159,"spinlock_wait_cycles":46336,"thread_start_us":477,"threads_started":1}
I20260620 02:58:49.022285 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:49.013769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45249 (local address 127.0.97.1:36573)
0620 02:58:49.014683 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:49.014707 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:49.014860 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:49.015227 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:49.015232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:49.015276 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:49.015811 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:49.015823 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.017191 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.017196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:49.019236 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:49.019249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.020929 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.020935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:49.021090 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:49.021115 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 02:58:49.021283 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 02:58:49.021409 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":109,"spinlock_wait_cycles":13440,"thread_start_us":575,"threads_started":1}
I20260620 02:58:49.022780 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf47d252ef54fa88c6c804c22bb3a01" candidate_uuid: "b184657dd82540f0b211ab8716d8a791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45fcaa7f8b0a41468dd77136041d11f7" is_pre_election: true
I20260620 02:58:49.023309 1240 raft_consensus.cc:2454] T aaf47d252ef54fa88c6c804c22bb3a01 P 45fcaa7f8b0a41468dd77136041d11f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b184657dd82540f0b211ab8716d8a791 in term 0.
I20260620 02:58:49.023270 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf47d252ef54fa88c6c804c22bb3a01" candidate_uuid: "b184657dd82540f0b211ab8716d8a791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12747e7f712843f4ba8d8a9ff24c9ee2" is_pre_election: true
I20260620 02:58:49.023789 1373 raft_consensus.cc:2454] T aaf47d252ef54fa88c6c804c22bb3a01 P 12747e7f712843f4ba8d8a9ff24c9ee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b184657dd82540f0b211ab8716d8a791 in term 0.
I20260620 02:58:49.024047 1041 leader_election.cc:302] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45fcaa7f8b0a41468dd77136041d11f7, b184657dd82540f0b211ab8716d8a791; no voters:
I20260620 02:58:49.024432 1442 raft_consensus.cc:2781] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:58:49.024559 1442 raft_consensus.cc:482] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:58:49.024632 1442 raft_consensus.cc:3037] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:49.026772 1442 raft_consensus.cc:504] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b184657dd82540f0b211ab8716d8a791" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34495 } } peers { permanent_uuid: "45fcaa7f8b0a41468dd77136041d11f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33251 } } peers { permanent_uuid: "12747e7f712843f4ba8d8a9ff24c9ee2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45249 } }
I20260620 02:58:49.027330 1442 leader_election.cc:288] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791 [CANDIDATE]: Term 1 election: Requested vote from peers 45fcaa7f8b0a41468dd77136041d11f7 (127.0.97.2:33251), 12747e7f712843f4ba8d8a9ff24c9ee2 (127.0.97.3:45249)
I20260620 02:58:49.027716 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf47d252ef54fa88c6c804c22bb3a01" candidate_uuid: "b184657dd82540f0b211ab8716d8a791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45fcaa7f8b0a41468dd77136041d11f7"
I20260620 02:58:49.027769 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf47d252ef54fa88c6c804c22bb3a01" candidate_uuid: "b184657dd82540f0b211ab8716d8a791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12747e7f712843f4ba8d8a9ff24c9ee2"
I20260620 02:58:49.027935 1373 raft_consensus.cc:3037] T aaf47d252ef54fa88c6c804c22bb3a01 P 12747e7f712843f4ba8d8a9ff24c9ee2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:49.027925 1240 raft_consensus.cc:3037] T aaf47d252ef54fa88c6c804c22bb3a01 P 45fcaa7f8b0a41468dd77136041d11f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:49.031107 1373 raft_consensus.cc:2454] T aaf47d252ef54fa88c6c804c22bb3a01 P 12747e7f712843f4ba8d8a9ff24c9ee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b184657dd82540f0b211ab8716d8a791 in term 1.
I20260620 02:58:49.031100 1240 raft_consensus.cc:2454] T aaf47d252ef54fa88c6c804c22bb3a01 P 45fcaa7f8b0a41468dd77136041d11f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b184657dd82540f0b211ab8716d8a791 in term 1.
I20260620 02:58:49.031503 1040 leader_election.cc:302] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12747e7f712843f4ba8d8a9ff24c9ee2, b184657dd82540f0b211ab8716d8a791; no voters:
I20260620 02:58:49.031862 1442 raft_consensus.cc:2781] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:58:49.032670 1442 raft_consensus.cc:686] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791 [term 1 LEADER]: Becoming Leader. State: Replica: b184657dd82540f0b211ab8716d8a791, State: Running, Role: LEADER
I20260620 02:58:49.033135 1442 consensus_queue.cc:237] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b184657dd82540f0b211ab8716d8a791" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34495 } } peers { permanent_uuid: "45fcaa7f8b0a41468dd77136041d11f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33251 } } peers { permanent_uuid: "12747e7f712843f4ba8d8a9ff24c9ee2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45249 } }
I20260620 02:58:49.037635 962 catalog_manager.cc:5495] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791 reported cstate change: term changed from 0 to 1, leader changed from <none> to b184657dd82540f0b211ab8716d8a791 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "b184657dd82540f0b211ab8716d8a791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b184657dd82540f0b211ab8716d8a791" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45fcaa7f8b0a41468dd77136041d11f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12747e7f712843f4ba8d8a9ff24c9ee2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45249 } health_report { overall_health: UNKNOWN } } }
Jun 20 02:58:49 dist-test-slave-f1j0 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:58:49.064469 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:49.050528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.0.97.1:34495)
0620 02:58:49.050707 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:49.050712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:49.051226 (+ 514us) server_negotiation.cc:407] Connection header received
0620 02:58:49.051356 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:49.051359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:49.051439 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:49.051520 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:49.052910 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.053698 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.054869 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.055241 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.055327 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:49.059449 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:49.059463 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:49.059478 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:49.059526 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:49.062330 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:49.062906 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:49.062913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:49.062915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:49.063053 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:49.063352 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:49.063355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:49.063357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:49.063683 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:49.063880 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:49.064078 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 02:58:49.064236 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 02:58:49.087482 1286 tablet.cc:2369] T aaf47d252ef54fa88c6c804c22bb3a01 P 45fcaa7f8b0a41468dd77136041d11f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:58:49.087533 1419 tablet.cc:2369] T aaf47d252ef54fa88c6c804c22bb3a01 P 12747e7f712843f4ba8d8a9ff24c9ee2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 02:58:49 dist-test-slave-f1j0 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924329, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260620 02:58:49.096781 1152 tablet.cc:2369] T aaf47d252ef54fa88c6c804c22bb3a01 P b184657dd82540f0b211ab8716d8a791: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 02:58:49 dist-test-slave-f1j0 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924329, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:58:49.110941 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:49.097229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.0.97.1:34495)
0620 02:58:49.097476 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:49.097483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:49.097825 (+ 342us) server_negotiation.cc:407] Connection header received
0620 02:58:49.097951 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:49.097955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:49.098037 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:49.098132 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:49.099373 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.100093 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.101294 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.101763 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.101860 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:49.105985 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:49.105992 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:49.105995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:49.106027 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:49.108584 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:49.109137 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:49.109142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:49.109144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:49.109262 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:49.109555 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:49.109558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:49.109560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:49.109898 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:49.110123 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:49.110385 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 02:58:49.110604 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 02:58:49 dist-test-slave-f1j0 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:58:49 dist-test-slave-f1j0 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:58:49.154313 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:49.140849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.0.97.1:34495)
0620 02:58:49.141067 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:49.141076 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:49.141400 (+ 324us) server_negotiation.cc:407] Connection header received
0620 02:58:49.141630 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:49.141634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:49.141714 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:49.141825 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:49.143135 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.143897 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.145130 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.145550 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.145740 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:49.149454 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:49.149459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:49.149462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:49.149493 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:49.151837 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:49.152446 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:49.152453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:49.152456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:49.152578 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:49.152942 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:49.152947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:49.152950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:49.153211 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:49.153459 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:49.153753 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 02:58:49.153956 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 02:58:49.156391 1066 tablet_service.cc:2908] Rejecting scan request for tablet aaf47d252ef54fa88c6c804c22bb3a01: Uninitialized: clean time has not yet been initialized
I20260620 02:58:49.159288 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 1023
I20260620 02:58:49.171427 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:49.160301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45052 (local address 127.0.97.1:34495)
0620 02:58:49.160503 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:49.160509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:49.160734 (+ 225us) server_negotiation.cc:407] Connection header received
0620 02:58:49.160943 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:49.160947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:49.161025 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:49.161108 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:49.162254 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.162926 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.164111 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.164477 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.164563 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:49.166697 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:49.166701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:49.166705 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:49.166731 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:49.169259 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:49.169860 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:49.169864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:49.169866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:49.169972 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:49.170269 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:49.170272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:49.170274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:49.170491 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:49.170675 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:49.170929 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 02:58:49.171066 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 02:58:49.172098 1146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:58:49.262161 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 1023
I20260620 02:58:49.279862 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 1157
Jun 20 02:58:49 dist-test-slave-f1j0 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:58:49.293924 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:49.280582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.0.97.2:33251)
0620 02:58:49.280753 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:49.280758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:49.280775 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:58:49.280935 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:49.280939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:49.280999 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:49.281082 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:49.282212 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.282936 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.284107 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.284471 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.284559 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:49.288498 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:49.288511 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:49.288527 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:49.288562 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:49.291444 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:49.292080 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:49.292087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:49.292090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:49.292206 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:49.292559 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:49.292562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:49.292564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:49.292920 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:49.293142 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:49.293358 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 02:58:49.293503 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:58:49.294617 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:58:49.381687 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 1157
I20260620 02:58:49.397850 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 1290
Jun 20 02:58:49 dist-test-slave-f1j0 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:58:49.412523 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:49.398619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57248 (local address 127.0.97.3:45249)
0620 02:58:49.398776 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:49.398781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:49.398798 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:58:49.398943 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:49.398947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:49.399046 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:49.399133 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:49.400305 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.401061 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.402387 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:49.402776 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:49.402879 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:49.407233 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:49.407248 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:49.407265 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:49.407303 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:49.410063 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:49.410675 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:49.410682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:49.410685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:49.410943 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:49.411282 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:49.411285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:49.411287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:49.411609 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:49.411807 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:49.412085 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 02:58:49.412252 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 02:58:49.413125 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:58:49.501386 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 1290
I20260620 02:58:49.517724 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 930
I20260620 02:58:49.518898 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:58:49.645056 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 930
2026-06-20T02:58:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4065 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:58:49 dist-test-slave-f1j0 krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:58:49 dist-test-slave-f1j0 krb5kdc[1466](info): set up 2 sockets
Jun 20 02:58:49 dist-test-slave-f1j0 krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260620 02:58:50.713522 388 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 02:58:50 dist-test-slave-f1j0 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:58:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:58:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:58:50.890939 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35375
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:58:51.203855 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:58:51.204128 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:58:51.204222 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:58:51.211176 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:58:51.211277 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 02:58:51.211318 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:58:51.211378 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:58:51.211427 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:58:51.211480 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:58:51.221074 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35375
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:58:51.222267 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:58:51.223486 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:58:51.234712 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:51.234629 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:51.234793 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:58:51.235437 1479 server_base.cc:1034] running on GCE node
I20260620 02:58:51.236193 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:58:51.237530 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:58:51.238807 1479 hybrid_clock.cc:648] HybridClock initialized: now 1781924331238744 us; error 40 us; skew 500 ppm
I20260620 02:58:51.239161 1479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:58:51 dist-test-slave-f1j0 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:58:51.242319 1479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:58:51.243723 1479 webserver.cc:460] Webserver started at http://127.0.97.62:42777/ using document root <none> and password file <none>
I20260620 02:58:51.244423 1479 fs_manager.cc:359] Metadata directory not provided
I20260620 02:58:51.244553 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:58:51.244887 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:58:51.247888 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "bbbcafcba8034a5a84449a352bfa8550"
format_stamp: "Formatted at 2026-06-20 02:58:51 on dist-test-slave-f1j0"
server_key: "f26942c238cfc2dcfd23b049e5247eea"
server_key_iv: "fd557bdfdeb2b120fd4e54c224bbd2c1"
server_key_version: "encryptionkey@0"
I20260620 02:58:51.248714 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "bbbcafcba8034a5a84449a352bfa8550"
format_stamp: "Formatted at 2026-06-20 02:58:51 on dist-test-slave-f1j0"
server_key: "f26942c238cfc2dcfd23b049e5247eea"
server_key_iv: "fd557bdfdeb2b120fd4e54c224bbd2c1"
server_key_version: "encryptionkey@0"
I20260620 02:58:51.253717 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:58:51.256992 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:51.258260 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:58:51.258440 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "bbbcafcba8034a5a84449a352bfa8550"
format_stamp: "Formatted at 2026-06-20 02:58:51 on dist-test-slave-f1j0"
server_key: "f26942c238cfc2dcfd23b049e5247eea"
server_key_iv: "fd557bdfdeb2b120fd4e54c224bbd2c1"
server_key_version: "encryptionkey@0"
I20260620 02:58:51.258657 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:58:51.276976 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:58:51.281351 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:58:51.281672 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:58:51.302778 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35375
I20260620 02:58:51.302796 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35375 every 8 connection(s)
I20260620 02:58:51.304180 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:58:51.307710 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 1479
I20260620 02:58:51.307915 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 02:58:51.308308 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:51.308804 388 external_mini_cluster.cc:1428] Setting key d84368e812e5e8f6d7099a63cf0e54c0
I20260620 02:58:51.322716 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550: Bootstrap starting.
Jun 20 02:58:51 dist-test-slave-f1j0 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:51.326428 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:51.327754 1556 log.cc:824] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550: Log is configured to *not* fsync() on all Append() calls
I20260620 02:58:51.331065 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550: No bootstrap required, opened a new log
I20260620 02:58:51.332544 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:51.310864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55250 (local address 127.0.97.62:35375)
0620 02:58:51.311637 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:51.311651 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:51.311710 (+ 59us) server_negotiation.cc:407] Connection header received
0620 02:58:51.313612 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:51.313641 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:51.314164 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:51.314619 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:51.316007 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:51.317957 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:51.319160 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:51.320239 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:51.320459 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:51.324308 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:51.324319 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:51.324338 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:51.324398 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:51.328170 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:51.328997 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:51.329003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:51.329009 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:51.329215 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:51.329558 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:51.329562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:51.329565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:51.330242 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:51.330516 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:51.330828 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 02:58:51.331235 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":307,"threads_started":1}
I20260620 02:58:51.339346 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbcafcba8034a5a84449a352bfa8550" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35375 } }
I20260620 02:58:51.339699 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:51.339818 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbbcafcba8034a5a84449a352bfa8550, State: Initialized, Role: FOLLOWER
I20260620 02:58:51.340354 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbcafcba8034a5a84449a352bfa8550" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35375 } }
I20260620 02:58:51.340544 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:58:51.340636 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:58:51.340760 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:51.343106 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbcafcba8034a5a84449a352bfa8550" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35375 } }
I20260620 02:58:51.343542 1556 leader_election.cc:302] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbbcafcba8034a5a84449a352bfa8550; no voters:
I20260620 02:58:51.344072 1556 leader_election.cc:288] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:58:51.344223 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:58:51.344897 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550 [term 1 LEADER]: Becoming Leader. State: Replica: bbbcafcba8034a5a84449a352bfa8550, State: Running, Role: LEADER
I20260620 02:58:51.345458 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbcafcba8034a5a84449a352bfa8550" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35375 } }
I20260620 02:58:51.346226 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:58:51.348496 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbbcafcba8034a5a84449a352bfa8550. Latest consensus state: current_term: 1 leader_uuid: "bbbcafcba8034a5a84449a352bfa8550" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbcafcba8034a5a84449a352bfa8550" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35375 } } }
I20260620 02:58:51.348932 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550 [sys.catalog]: This master's current role is: LEADER
I20260620 02:58:51.348489 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbbcafcba8034a5a84449a352bfa8550" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbcafcba8034a5a84449a352bfa8550" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35375 } } }
I20260620 02:58:51.349432 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550 [sys.catalog]: This master's current role is: LEADER
I20260620 02:58:51.350679 1566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:58:51.359071 1566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:58:51.369482 1566 catalog_manager.cc:1347] Generated new cluster ID: 60bdd87d2bbc4503af5f4df4d77e4149
I20260620 02:58:51.369622 1566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:58:51.396488 1566 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:58:51.397881 1566 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:58:51.404531 1566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550: Generated new TSK 0
I20260620 02:58:51.405447 1566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:58:51.450287 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35375
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:58:51.774169 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:58:51.774505 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:58:51.774672 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:58:51.782060 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:58:51.782225 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:58:51.782367 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:58:51.782502 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:58:51.792410 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:58:51.793781 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:58:51.795089 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:58:51.808014 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:51.808061 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:51.808099 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:58:51.808974 1582 server_base.cc:1034] running on GCE node
I20260620 02:58:51.809527 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:58:51.810448 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:58:51.811650 1582 hybrid_clock.cc:648] HybridClock initialized: now 1781924331811570 us; error 54 us; skew 500 ppm
I20260620 02:58:51.811933 1582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:58:51 dist-test-slave-f1j0 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:58:51.815003 1582 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:58:51.816524 1582 webserver.cc:460] Webserver started at http://127.0.97.1:45307/ using document root <none> and password file <none>
I20260620 02:58:51.817203 1582 fs_manager.cc:359] Metadata directory not provided
I20260620 02:58:51.817312 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:58:51.817698 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:58:51.820503 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "c1b7439223404fe6a73b86c5a0b8abcf"
format_stamp: "Formatted at 2026-06-20 02:58:51 on dist-test-slave-f1j0"
server_key: "713f7c27a64262a35d359e9d29c3d02a"
server_key_iv: "40ab414d69b673727531d78fc70ac22b"
server_key_version: "encryptionkey@0"
I20260620 02:58:51.821264 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c1b7439223404fe6a73b86c5a0b8abcf"
format_stamp: "Formatted at 2026-06-20 02:58:51 on dist-test-slave-f1j0"
server_key: "713f7c27a64262a35d359e9d29c3d02a"
server_key_iv: "40ab414d69b673727531d78fc70ac22b"
server_key_version: "encryptionkey@0"
I20260620 02:58:51.826316 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 02:58:51.829788 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:51.831110 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:58:51.831292 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "c1b7439223404fe6a73b86c5a0b8abcf"
format_stamp: "Formatted at 2026-06-20 02:58:51 on dist-test-slave-f1j0"
server_key: "713f7c27a64262a35d359e9d29c3d02a"
server_key_iv: "40ab414d69b673727531d78fc70ac22b"
server_key_version: "encryptionkey@0"
I20260620 02:58:51.831502 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:58:51.848753 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:58:51.853031 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:58:51.853307 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:58:51.855778 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:58:51.855852 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:51.856005 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:58:51.856066 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:58:51 dist-test-slave-f1j0 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:51.880301 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:51.856505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51981 (local address 127.0.97.62:35375)
0620 02:58:51.856934 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:51.856942 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:51.858394 (+ 1452us) server_negotiation.cc:407] Connection header received
0620 02:58:51.861005 (+ 2611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:51.861010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:51.861110 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:51.861248 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:51.863991 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:51.865237 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:51.867618 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:51.868402 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:51.868564 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:51.873172 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:51.873178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:51.873181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:51.873214 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:51.876330 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:51.877146 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:51.877153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:51.877160 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:51.877331 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:51.877979 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:51.877987 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:51.877992 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:51.878400 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:51.878577 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:51.879545 (+ 968us) server_negotiation.cc:299] Negotiation successful
0620 02:58:51.879816 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":228,"threads_started":1}
I20260620 02:58:51.881825 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:51.857124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35375 (local address 127.0.97.1:51981)
0620 02:58:51.858212 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:51.858259 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:51.860685 (+ 2426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:51.861609 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:51.861622 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:51.862598 (+ 976us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:51.863716 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:51.863744 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:51.865455 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:51.865462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:51.867444 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:51.867458 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:51.868688 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:51.868696 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:51.868913 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:51.869839 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:51.869881 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:51.872935 (+ 3054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:51.876461 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:51.876470 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:51.876489 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:51.876992 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:51.877508 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:51.877515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:51.877520 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:51.877843 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:51.878556 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:51.878567 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:51.879046 (+ 479us) client_negotiation.cc:769] Sending connection context
0620 02:58:51.879412 (+ 366us) client_negotiation.cc:240] Negotiation successful
0620 02:58:51.880077 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":578,"threads_started":1}
I20260620 02:58:51.901674 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40209
I20260620 02:58:51.901703 1717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40209 every 8 connection(s)
I20260620 02:58:51.902928 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:58:51.905650 1718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35375
I20260620 02:58:51.905898 1718 heartbeater.cc:464] Registering TS with master...
I20260620 02:58:51.906697 1718 heartbeater.cc:511] Master 127.0.97.62:35375 requested a full tablet report, sending...
I20260620 02:58:51.908608 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:51.889841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40077 (local address 127.0.97.62:35375)
0620 02:58:51.889998 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:51.890003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:51.895228 (+ 5225us) server_negotiation.cc:407] Connection header received
0620 02:58:51.895548 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:51.895555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:51.895623 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:51.895737 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:51.896640 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:51.897748 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:51.899694 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:51.900360 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:51.900511 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:51.902535 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:51.902541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:51.902546 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:51.902587 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:51.905878 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:51.906550 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:51.906557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:51.906561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:51.906703 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:51.907108 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:51.907112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:51.907118 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:51.907402 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:51.907551 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:51.908035 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 02:58:51.908261 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:58:51.908641 1702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:51.893959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35375 (local address 127.0.97.1:40077)
0620 02:58:51.895067 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:51.895112 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:51.895367 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:51.895765 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:51.895770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:51.896225 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:51.896506 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:51.896514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:51.897951 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:51.897961 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:51.899558 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:51.899569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:51.900557 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:51.900564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:51.900748 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:51.901454 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:51.901470 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:51.902374 (+ 904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:51.906029 (+ 3655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:51.906034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:51.906038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:51.906433 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:51.906840 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:51.906846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:51.906849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:51.906999 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:51.907521 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:51.907525 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:51.907747 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 02:58:51.907893 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 02:58:51.908123 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"thread_start_us":381,"threads_started":1}
I20260620 02:58:51.908632 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 1582
I20260620 02:58:51.908802 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:58:51.908999 1511 ts_manager.cc:194] Registered new tserver with Master: c1b7439223404fe6a73b86c5a0b8abcf (127.0.97.1:40209)
I20260620 02:58:51.909562 388 external_mini_cluster.cc:1428] Setting key 5b15560d8c684889771fb4b703e9fa00
I20260620 02:58:51.910914 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51981
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:58:51.948283 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35375
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:58:52.265540 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:58:52.265870 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:58:52.265965 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:58:52.273034 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:58:52.273145 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:58:52.273229 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:58:52.273307 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:58:52.282902 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:58:52.284278 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:58:52.285698 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:58:52.298969 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:52.298929 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:52.298956 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:58:52.299984 1722 server_base.cc:1034] running on GCE node
I20260620 02:58:52.300614 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:58:52.301445 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:58:52.302681 1722 hybrid_clock.cc:648] HybridClock initialized: now 1781924332302594 us; error 57 us; skew 500 ppm
I20260620 02:58:52.302994 1722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:58:52 dist-test-slave-f1j0 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:58:52.306576 1722 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:58:52.308091 1722 webserver.cc:460] Webserver started at http://127.0.97.2:38019/ using document root <none> and password file <none>
I20260620 02:58:52.308781 1722 fs_manager.cc:359] Metadata directory not provided
I20260620 02:58:52.308912 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:58:52.309239 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:58:52.312063 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "60260c3e799d472891e300768983b20d"
format_stamp: "Formatted at 2026-06-20 02:58:52 on dist-test-slave-f1j0"
server_key: "8659c49c2112a6082d67449fb9af35ef"
server_key_iv: "4358ce1a43bf3a1b361c9918195be8e7"
server_key_version: "encryptionkey@0"
I20260620 02:58:52.312827 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "60260c3e799d472891e300768983b20d"
format_stamp: "Formatted at 2026-06-20 02:58:52 on dist-test-slave-f1j0"
server_key: "8659c49c2112a6082d67449fb9af35ef"
server_key_iv: "4358ce1a43bf3a1b361c9918195be8e7"
server_key_version: "encryptionkey@0"
I20260620 02:58:52.318010 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260620 02:58:52.321172 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:52.322476 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:58:52.322644 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "60260c3e799d472891e300768983b20d"
format_stamp: "Formatted at 2026-06-20 02:58:52 on dist-test-slave-f1j0"
server_key: "8659c49c2112a6082d67449fb9af35ef"
server_key_iv: "4358ce1a43bf3a1b361c9918195be8e7"
server_key_version: "encryptionkey@0"
I20260620 02:58:52.322857 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:58:52.356366 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:58:52.360635 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:58:52.360913 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:58:52.363626 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:58:52.363745 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:52.363909 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:58:52.363983 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:58:52 dist-test-slave-f1j0 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:52.389009 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:52.365119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35375 (local address 127.0.97.2:46511)
0620 02:58:52.366087 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:52.366152 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:52.369135 (+ 2983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:52.370073 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:52.370089 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:52.371072 (+ 983us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:52.372127 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:52.372157 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:52.373429 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:52.373436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:52.375134 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:52.375148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:52.376174 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:52.376182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:52.376379 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:52.377040 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:52.377060 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:52.380070 (+ 3010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:52.384487 (+ 4417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:52.384494 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:52.384507 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:52.384910 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:52.385439 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:52.385443 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:52.385445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:52.385694 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:52.386365 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:52.386372 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:52.386887 (+ 515us) client_negotiation.cc:769] Sending connection context
0620 02:58:52.387233 (+ 346us) client_negotiation.cc:240] Negotiation successful
0620 02:58:52.387771 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":401,"threads_started":1}
I20260620 02:58:52.389607 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:52.364444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46511 (local address 127.0.97.62:35375)
0620 02:58:52.364628 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:52.364634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:52.366282 (+ 1648us) server_negotiation.cc:407] Connection header received
0620 02:58:52.369499 (+ 3217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:52.369507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:52.369626 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:52.369757 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:52.372332 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:52.373215 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:52.375312 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:52.375926 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:52.376051 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:52.380329 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:52.380337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:52.380343 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:52.380394 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:52.384339 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:52.385041 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:52.385049 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:52.385055 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:52.385250 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:52.385818 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:52.385824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:52.385827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:52.386238 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:52.386376 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:52.389040 (+ 2664us) server_negotiation.cc:299] Negotiation successful
0620 02:58:52.389253 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:58:52.412410 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45167
I20260620 02:58:52.412910 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45167 every 8 connection(s)
I20260620 02:58:52.413914 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:58:52.415064 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 1722
I20260620 02:58:52.415231 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:58:52.415784 388 external_mini_cluster.cc:1428] Setting key ac73eeb60b388c22074d6eb593851fc5
I20260620 02:58:52.416169 1857 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35375
I20260620 02:58:52.416427 1857 heartbeater.cc:464] Registering TS with master...
I20260620 02:58:52.418277 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:52.397280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42393 (local address 127.0.97.62:35375)
0620 02:58:52.397441 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:52.397445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:52.402556 (+ 5111us) server_negotiation.cc:407] Connection header received
0620 02:58:52.402754 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:52.402761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:52.402855 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:52.402965 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:52.404144 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:52.405502 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:52.407549 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:52.408134 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:52.408288 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:52.411151 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:52.411160 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:52.411167 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:52.411214 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:52.414275 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:52.415544 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:52.415554 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:52.415560 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:52.415784 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:52.416413 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:52.416421 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:52.416427 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:52.416991 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:52.417184 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:52.417751 (+ 567us) server_negotiation.cc:299] Negotiation successful
0620 02:58:52.417965 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 02:58:52.419283 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:52.397429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35375 (local address 127.0.97.2:42393)
0620 02:58:52.402427 (+ 4998us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:52.402449 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:52.402606 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:52.403119 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:52.403124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:52.403560 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:52.403953 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:52.403971 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:52.405695 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:52.405702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:52.407387 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:52.407398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:52.408398 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:52.408413 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:52.408665 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:52.409682 (+ 1017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:52.409709 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:52.410935 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:52.414455 (+ 3520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:52.414460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:52.414469 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:52.415277 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:52.416009 (+ 732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:52.416016 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:52.416023 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:52.416216 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:52.417142 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:52.417147 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:52.417340 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 02:58:52.418435 (+ 1095us) client_negotiation.cc:240] Negotiation successful
0620 02:58:52.418777 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4848,"thread_start_us":4729,"threads_started":1}
I20260620 02:58:52.422111 1857 heartbeater.cc:511] Master 127.0.97.62:35375 requested a full tablet report, sending...
I20260620 02:58:52.423580 1511 ts_manager.cc:194] Registered new tserver with Master: 60260c3e799d472891e300768983b20d (127.0.97.2:45167)
I20260620 02:58:52.424341 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46511
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:58:52.456338 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35375
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:58:52.767740 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:58:52.768097 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:58:52.768278 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:58:52.775722 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:58:52.775871 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:58:52.776024 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:58:52.776163 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:58:52.786293 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:58:52.787588 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:58:52.788932 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:58:52.801559 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:52.801632 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:52.801635 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:58:52.802446 1861 server_base.cc:1034] running on GCE node
I20260620 02:58:52.803045 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:58:52.803954 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:58:52.805188 1861 hybrid_clock.cc:648] HybridClock initialized: now 1781924332805124 us; error 42 us; skew 500 ppm
I20260620 02:58:52.805475 1861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:58:52 dist-test-slave-f1j0 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:58:52.808554 1861 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:58:52.810041 1861 webserver.cc:460] Webserver started at http://127.0.97.3:35009/ using document root <none> and password file <none>
I20260620 02:58:52.810714 1861 fs_manager.cc:359] Metadata directory not provided
I20260620 02:58:52.810832 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:58:52.811152 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:58:52.814049 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "1d68e73fdd1142f9972b53d81b67e035"
format_stamp: "Formatted at 2026-06-20 02:58:52 on dist-test-slave-f1j0"
server_key: "b2409b0377d88927063e17efe8e94675"
server_key_iv: "766269ee1d05a6d55681f272b067ee57"
server_key_version: "encryptionkey@0"
I20260620 02:58:52.814853 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1d68e73fdd1142f9972b53d81b67e035"
format_stamp: "Formatted at 2026-06-20 02:58:52 on dist-test-slave-f1j0"
server_key: "b2409b0377d88927063e17efe8e94675"
server_key_iv: "766269ee1d05a6d55681f272b067ee57"
server_key_version: "encryptionkey@0"
I20260620 02:58:52.820005 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 02:58:52.823590 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:52.824932 1861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:58:52.825099 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "1d68e73fdd1142f9972b53d81b67e035"
format_stamp: "Formatted at 2026-06-20 02:58:52 on dist-test-slave-f1j0"
server_key: "b2409b0377d88927063e17efe8e94675"
server_key_iv: "766269ee1d05a6d55681f272b067ee57"
server_key_version: "encryptionkey@0"
I20260620 02:58:52.825322 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:58:52.855336 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:58:52.859772 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:58:52.860057 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:58:52.862965 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:58:52.863073 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:52.863282 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:58:52.863363 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:58:52 dist-test-slave-f1j0 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:52.887580 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:52.863320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41211 (local address 127.0.97.62:35375)
0620 02:58:52.863528 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:52.863534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:52.864971 (+ 1437us) server_negotiation.cc:407] Connection header received
0620 02:58:52.867128 (+ 2157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:52.867135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:52.867239 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:52.867358 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:52.869644 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:52.870996 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:52.873684 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:52.874352 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:52.874522 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:52.879542 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:52.879547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:52.879551 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:52.879585 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:52.883694 (+ 4109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:52.884807 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:52.884817 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:52.884820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:52.884988 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:52.885454 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:52.885459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:52.885463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:52.885845 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:52.886021 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:52.886985 (+ 964us) server_negotiation.cc:299] Negotiation successful
0620 02:58:52.887219 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:58:52.889063 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:52.863899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35375 (local address 127.0.97.3:41211)
0620 02:58:52.864790 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:52.864850 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:52.866842 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:52.867660 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:52.867671 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:52.868397 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:52.869369 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:52.869420 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:52.871180 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:52.871186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:52.873498 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:52.873520 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:52.874632 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:52.874646 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:52.874890 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:52.875804 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:52.875829 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:52.879291 (+ 3462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:52.883995 (+ 4704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:52.884007 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:52.884023 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:52.884664 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:52.885103 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:52.885107 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:52.885110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:52.885329 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:52.886004 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:52.886013 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:52.886483 (+ 470us) client_negotiation.cc:769] Sending connection context
0620 02:58:52.886862 (+ 379us) client_negotiation.cc:240] Negotiation successful
0620 02:58:52.887486 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":399,"threads_started":1}
I20260620 02:58:52.911566 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:52.895904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57695 (local address 127.0.97.62:35375)
0620 02:58:52.896103 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:52.896110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:52.897083 (+ 973us) server_negotiation.cc:407] Connection header received
0620 02:58:52.897308 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:52.897315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:52.897416 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:52.897549 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:52.898503 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:52.899222 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:52.900711 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:52.901322 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:52.901452 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:52.904284 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:52.904303 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:52.904307 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:52.904346 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:52.908450 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:52.909361 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:52.909370 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:52.909375 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:52.909532 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:52.910012 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:52.910017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:52.910021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:52.910302 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:52.910452 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:52.910957 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 02:58:52.911172 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:58:52.911664 1960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:52.895937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35375 (local address 127.0.97.3:57695)
0620 02:58:52.896941 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:52.896967 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:52.897168 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:52.897598 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:52.897603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:52.898074 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:52.898353 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:52.898361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:52.899356 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:52.899362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:52.900530 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:52.900544 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:52.901876 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:52.901886 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:52.902100 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:52.902855 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:52.902880 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:52.904054 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:52.908610 (+ 4556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:52.908614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:52.908616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:52.909221 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:52.909706 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:52.909716 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:52.909722 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:52.909876 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:52.910421 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:52.910426 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:52.910659 (+ 233us) client_negotiation.cc:769] Sending connection context
0620 02:58:52.910837 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 02:58:52.911185 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":597,"threads_started":1}
I20260620 02:58:52.913929 1718 heartbeater.cc:503] Master 127.0.97.62:35375 was elected leader, sending a full tablet report...
I20260620 02:58:52.918764 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33023
I20260620 02:58:52.918784 1995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33023 every 8 connection(s)
I20260620 02:58:52.919955 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:58:52.921907 1996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35375
I20260620 02:58:52.922137 1996 heartbeater.cc:464] Registering TS with master...
I20260620 02:58:52.922739 1996 heartbeater.cc:511] Master 127.0.97.62:35375 requested a full tablet report, sending...
I20260620 02:58:52.923991 1511 ts_manager.cc:194] Registered new tserver with Master: 1d68e73fdd1142f9972b53d81b67e035 (127.0.97.3:33023)
I20260620 02:58:52.924793 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41211
I20260620 02:58:52.924984 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 1861
I20260620 02:58:52.925156 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:58:52.925623 388 external_mini_cluster.cc:1428] Setting key 986ab1295df2a30d2c143dc5c2c36c5f
I20260620 02:58:52.928406 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:58:52 dist-test-slave-f1j0 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924332, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:58:52 dist-test-slave-f1j0 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924332, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:52.972126 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:52.958715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55266 (local address 127.0.97.62:35375)
0620 02:58:52.958947 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:52.958956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:52.959098 (+ 142us) server_negotiation.cc:407] Connection header received
0620 02:58:52.959251 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:52.959255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:52.959322 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:52.959410 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:52.960697 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:52.961433 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:52.962820 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:52.963221 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:52.963313 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:52.967008 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:52.967013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:52.967016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:52.967048 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:52.969360 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:52.970002 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:52.970006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:52.970009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:52.970118 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:52.970542 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:52.970546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:52.970557 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:52.970862 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:52.971100 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:52.971478 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 02:58:52.971710 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 02:58:52.976576 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:58:52.983501 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:58:53.007644 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:52.995908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57498 (local address 127.0.97.3:33023)
0620 02:58:52.999743 (+ 3835us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:52.999751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:52.999782 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:58:52.999875 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:52.999884 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:53.000104 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:53.000681 (+ 577us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:53.001014 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.002378 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.005236 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.006517 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.006677 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.006917 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:53.007139 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 02:58:53.007297 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3644,"thread_start_us":1875,"threads_started":1}
I20260620 02:58:53.007740 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:52.995837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33023 (local address 127.0.0.1:57498)
0620 02:58:52.998460 (+ 2623us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:52.998518 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:52.998687 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:53.000311 (+ 1624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:53.000318 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:53.000375 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:53.000837 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.000850 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.002545 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.002551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.005056 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.005070 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.006705 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.006711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.006891 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.006946 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 02:58:53.007100 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 02:58:53.007295 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2365,"mutex_wait_us":59,"spinlock_wait_cycles":896,"thread_start_us":2243,"threads_started":1}
I20260620 02:58:53.008455 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:52.998287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40209 (local address 127.0.0.1:58192)
0620 02:58:52.999361 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:52.999377 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:52.999493 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:53.001331 (+ 1838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:53.001335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:53.001359 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:53.001805 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.001816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.003674 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.003679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.005791 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.005801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.007614 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.007620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.007802 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.007836 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 02:58:53.007899 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 02:58:53.008041 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":952,"mutex_wait_us":69,"thread_start_us":813,"threads_started":1}
I20260620 02:58:53.009261 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:52.995866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45167 (local address 127.0.0.1:58502)
0620 02:58:52.999194 (+ 3328us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:52.999216 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:52.999343 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:53.000319 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:53.000322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:53.000385 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:53.001956 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.001969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.003712 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.003720 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.005936 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.005951 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.008479 (+ 2528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.008486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.008638 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.008667 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 02:58:53.008742 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 02:58:53.008869 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3197,"mutex_wait_us":1110,"spinlock_wait_cycles":52864,"thread_start_us":3351,"threads_started":1}
I20260620 02:58:53.009407 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:52.998850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58192 (local address 127.0.97.1:40209)
0620 02:58:53.000930 (+ 2080us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:53.000935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:53.000955 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:58:53.001029 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:53.001034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:53.001164 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:53.001347 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:53.001965 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.003515 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.005938 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.007391 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.008672 (+ 1281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.008869 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:53.008943 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 02:58:53.009104 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1946,"thread_start_us":1827,"threads_started":1}
I20260620 02:58:53.009950 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:52.997071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58502 (local address 127.0.97.2:45167)
0620 02:58:52.997754 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:52.997767 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:52.999635 (+ 1868us) server_negotiation.cc:407] Connection header received
0620 02:58:52.999768 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:52.999779 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:53.000103 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:53.000359 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:53.002134 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.003562 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.006096 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.008257 (+ 2161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.008390 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.008607 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:53.009486 (+ 879us) server_negotiation.cc:299] Negotiation successful
0620 02:58:53.009675 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":296,"threads_started":1}
I20260620 02:58:53.011961 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 242d966178a940cc91cc48fbcd8d98ae (DEFAULT_TABLE table=test-table [id=19c7a2938528482bae60e811a3e5e928]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:58:53.012895 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 242d966178a940cc91cc48fbcd8d98ae (DEFAULT_TABLE table=test-table [id=19c7a2938528482bae60e811a3e5e928]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:58:53.012809 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 242d966178a940cc91cc48fbcd8d98ae (DEFAULT_TABLE table=test-table [id=19c7a2938528482bae60e811a3e5e928]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:58:53.013513 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242d966178a940cc91cc48fbcd8d98ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:53.013990 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242d966178a940cc91cc48fbcd8d98ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:53.014108 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242d966178a940cc91cc48fbcd8d98ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:53.025422 2015 tablet_bootstrap.cc:492] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf: Bootstrap starting.
I20260620 02:58:53.029086 2015 tablet_bootstrap.cc:654] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:53.029280 2017 tablet_bootstrap.cc:492] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035: Bootstrap starting.
I20260620 02:58:53.030402 2016 tablet_bootstrap.cc:492] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d: Bootstrap starting.
I20260620 02:58:53.030452 2015 log.cc:824] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf: Log is configured to *not* fsync() on all Append() calls
I20260620 02:58:53.033352 2016 tablet_bootstrap.cc:654] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:53.033555 2017 tablet_bootstrap.cc:654] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:53.035125 2016 log.cc:824] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d: Log is configured to *not* fsync() on all Append() calls
I20260620 02:58:53.035187 2017 log.cc:824] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035: Log is configured to *not* fsync() on all Append() calls
I20260620 02:58:53.036720 2015 tablet_bootstrap.cc:492] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf: No bootstrap required, opened a new log
I20260620 02:58:53.037038 2015 ts_tablet_manager.cc:1397] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260620 02:58:53.037987 2017 tablet_bootstrap.cc:492] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035: No bootstrap required, opened a new log
I20260620 02:58:53.038055 2016 tablet_bootstrap.cc:492] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d: No bootstrap required, opened a new log
I20260620 02:58:53.038245 2017 ts_tablet_manager.cc:1397] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 02:58:53.038308 2016 ts_tablet_manager.cc:1397] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260620 02:58:53.045768 2017 raft_consensus.cc:348] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } }
I20260620 02:58:53.046254 2017 raft_consensus.cc:374] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:53.046355 2017 raft_consensus.cc:729] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d68e73fdd1142f9972b53d81b67e035, State: Initialized, Role: FOLLOWER
I20260620 02:58:53.047086 2017 consensus_queue.cc:260] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } }
I20260620 02:58:53.047428 2015 raft_consensus.cc:348] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } }
I20260620 02:58:53.047883 2015 raft_consensus.cc:374] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:53.048046 2015 raft_consensus.cc:729] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b7439223404fe6a73b86c5a0b8abcf, State: Initialized, Role: FOLLOWER
I20260620 02:58:53.048177 1996 heartbeater.cc:503] Master 127.0.97.62:35375 was elected leader, sending a full tablet report...
I20260620 02:58:53.048810 2015 consensus_queue.cc:260] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } }
I20260620 02:58:53.048841 2016 raft_consensus.cc:348] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } }
I20260620 02:58:53.049268 2016 raft_consensus.cc:374] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:53.049319 2017 ts_tablet_manager.cc:1428] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260620 02:58:53.049463 2016 raft_consensus.cc:729] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60260c3e799d472891e300768983b20d, State: Initialized, Role: FOLLOWER
I20260620 02:58:53.050338 2016 consensus_queue.cc:260] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } }
I20260620 02:58:53.054598 1857 heartbeater.cc:503] Master 127.0.97.62:35375 was elected leader, sending a full tablet report...
I20260620 02:58:53.055197 2016 ts_tablet_manager.cc:1428] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d: Time spent starting tablet: real 0.017s user 0.007s sys 0.007s
I20260620 02:58:53.056762 2015 ts_tablet_manager.cc:1428] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf: Time spent starting tablet: real 0.020s user 0.015s sys 0.003s
W20260620 02:58:53.155911 1719 tablet.cc:2369] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:58:53.168774 1858 tablet.cc:2369] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:58:53.171435 1997 tablet.cc:2369] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:58:53.322952 2023 raft_consensus.cc:482] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:58:53.323230 2023 raft_consensus.cc:504] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } }
I20260620 02:58:53.324873 2023 leader_election.cc:288] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d68e73fdd1142f9972b53d81b67e035 (127.0.97.3:33023), c1b7439223404fe6a73b86c5a0b8abcf (127.0.97.1:40209)
I20260620 02:58:53.333930 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.325366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50457 (local address 127.0.97.1:40209)
0620 02:58:53.325535 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:53.325541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:53.326326 (+ 785us) server_negotiation.cc:407] Connection header received
0620 02:58:53.326519 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:53.326525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:53.326613 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:53.326716 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:53.327461 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.328552 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.331591 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.332822 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.332962 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.333104 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:53.333516 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 02:58:53.333664 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 02:58:53.333930 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.325311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57167 (local address 127.0.97.3:33023)
0620 02:58:53.325502 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:53.325508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:53.326326 (+ 818us) server_negotiation.cc:407] Connection header received
0620 02:58:53.326519 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:53.326525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:53.326603 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:53.326716 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:53.327461 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.328552 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.331591 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.332823 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.332962 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.333073 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:53.333516 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 02:58:53.333664 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:58:53.334219 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.325239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33023 (local address 127.0.97.2:57167)
0620 02:58:53.326219 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:53.326236 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:53.326360 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:53.326742 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:53.326747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:53.326801 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:53.327326 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.327335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.328699 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.328705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.331415 (+ 2710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.331429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.333091 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.333095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.333295 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.333390 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 02:58:53.333477 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 02:58:53.333716 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":260,"spinlock_wait_cycles":40192,"thread_start_us":505,"threads_started":1}
I20260620 02:58:53.334263 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.325130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40209 (local address 127.0.97.2:50457)
0620 02:58:53.326169 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:53.326200 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:53.326391 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:53.326742 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:53.326747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:53.326781 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:53.327290 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.327305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.328708 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.328713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.331419 (+ 2706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.331429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.333079 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.333086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.333298 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.333382 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 02:58:53.333477 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 02:58:53.333716 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"mutex_wait_us":294,"spinlock_wait_cycles":23424,"thread_start_us":644,"threads_started":1}
I20260620 02:58:53.335358 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242d966178a940cc91cc48fbcd8d98ae" candidate_uuid: "60260c3e799d472891e300768983b20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d68e73fdd1142f9972b53d81b67e035" is_pre_election: true
I20260620 02:58:53.335542 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242d966178a940cc91cc48fbcd8d98ae" candidate_uuid: "60260c3e799d472891e300768983b20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" is_pre_election: true
I20260620 02:58:53.336054 1668 raft_consensus.cc:2454] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60260c3e799d472891e300768983b20d in term 0.
I20260620 02:58:53.336297 1946 raft_consensus.cc:2454] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60260c3e799d472891e300768983b20d in term 0.
I20260620 02:58:53.336782 1739 leader_election.cc:302] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60260c3e799d472891e300768983b20d, c1b7439223404fe6a73b86c5a0b8abcf; no voters:
I20260620 02:58:53.337231 2023 raft_consensus.cc:2781] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:58:53.337373 2023 raft_consensus.cc:482] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:58:53.337468 2023 raft_consensus.cc:3037] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:53.339572 2023 raft_consensus.cc:504] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } }
I20260620 02:58:53.340181 2023 leader_election.cc:288] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [CANDIDATE]: Term 1 election: Requested vote from peers 1d68e73fdd1142f9972b53d81b67e035 (127.0.97.3:33023), c1b7439223404fe6a73b86c5a0b8abcf (127.0.97.1:40209)
I20260620 02:58:53.340687 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242d966178a940cc91cc48fbcd8d98ae" candidate_uuid: "60260c3e799d472891e300768983b20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d68e73fdd1142f9972b53d81b67e035"
I20260620 02:58:53.340699 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242d966178a940cc91cc48fbcd8d98ae" candidate_uuid: "60260c3e799d472891e300768983b20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b7439223404fe6a73b86c5a0b8abcf"
I20260620 02:58:53.340857 1946 raft_consensus.cc:3037] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:53.340992 1668 raft_consensus.cc:3037] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:53.343045 1946 raft_consensus.cc:2454] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60260c3e799d472891e300768983b20d in term 1.
I20260620 02:58:53.343165 1668 raft_consensus.cc:2454] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60260c3e799d472891e300768983b20d in term 1.
I20260620 02:58:53.343501 1740 leader_election.cc:302] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d68e73fdd1142f9972b53d81b67e035, 60260c3e799d472891e300768983b20d; no voters:
I20260620 02:58:53.343833 2023 raft_consensus.cc:2781] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:58:53.344224 2023 raft_consensus.cc:686] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [term 1 LEADER]: Becoming Leader. State: Replica: 60260c3e799d472891e300768983b20d, State: Running, Role: LEADER
I20260620 02:58:53.344695 2023 consensus_queue.cc:237] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } }
I20260620 02:58:53.349514 1511 catalog_manager.cc:5495] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d reported cstate change: term changed from 0 to 1, leader changed from <none> to 60260c3e799d472891e300768983b20d (127.0.97.2). New cstate: current_term: 1 leader_uuid: "60260c3e799d472891e300768983b20d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20260620 02:58:53.382663 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.377711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.0.97.2:45167)
0620 02:58:53.377886 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:53.377891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:53.377931 (+ 40us) server_negotiation.cc:407] Connection header received
0620 02:58:53.378225 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:53.378229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:53.378308 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:53.378425 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:58:53.378753 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.379474 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.380726 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.381099 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.381191 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.381663 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:58:53.382161 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:58:53.382291 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:53.382405 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 02:58:53.382465 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 02:58:53.390880 1946 raft_consensus.cc:1261] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035 [term 1 FOLLOWER]: Refusing update from remote peer 60260c3e799d472891e300768983b20d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:58:53.390880 1668 raft_consensus.cc:1261] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf [term 1 FOLLOWER]: Refusing update from remote peer 60260c3e799d472891e300768983b20d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:58:53.391980 2023 consensus_queue.cc:1035] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:58:53.392318 2026 consensus_queue.cc:1035] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:58:53.406683 2030 mvcc.cc:205] Tried to move back new op lower bound from 7298762069559132160 to 7298762069383716864. Current Snapshot: MvccSnapshot[applied={T|T < 7298762069559132160}]
I20260620 02:58:53.422019 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:58:53.422356 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:58:53.425858 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550: Sending DeleteTablet for 3 replicas of tablet 242d966178a940cc91cc48fbcd8d98ae
I20260620 02:58:53.427011 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 242d966178a940cc91cc48fbcd8d98ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:58:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57498
I20260620 02:58:53.427091 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 242d966178a940cc91cc48fbcd8d98ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:58:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58192
I20260620 02:58:53.428112 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 242d966178a940cc91cc48fbcd8d98ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:58:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58502
I20260620 02:58:53.430979 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:58:53.432433 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:58:53.445214 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 1185225a64b74e6bacf7e2a5dcfc8e6c (DEFAULT_TABLE table=test-table [id=2b04294be61547bfb893a7c64553d5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:58:53.445834 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1185225a64b74e6bacf7e2a5dcfc8e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:53.445818 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 1185225a64b74e6bacf7e2a5dcfc8e6c (DEFAULT_TABLE table=test-table [id=2b04294be61547bfb893a7c64553d5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:58:53.446487 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1185225a64b74e6bacf7e2a5dcfc8e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:53.450019 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 1185225a64b74e6bacf7e2a5dcfc8e6c (DEFAULT_TABLE table=test-table [id=2b04294be61547bfb893a7c64553d5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:58:53.450451 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1185225a64b74e6bacf7e2a5dcfc8e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:53.455508 2016 tablet_bootstrap.cc:492] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d: Bootstrap starting.
I20260620 02:58:53.455509 2041 tablet_replica.cc:330] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d: stopping tablet replica
I20260620 02:58:53.455996 2041 raft_consensus.cc:2229] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:58:53.456892 2041 raft_consensus.cc:2258] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:58:53.457260 2016 tablet_bootstrap.cc:654] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:53.457862 2017 tablet_bootstrap.cc:492] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035: Bootstrap starting.
I20260620 02:58:53.459452 2017 tablet_bootstrap.cc:654] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:53.459626 2041 ts_tablet_manager.cc:1905] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:58:53.461606 2015 tablet_bootstrap.cc:492] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf: Bootstrap starting.
I20260620 02:58:53.463203 2015 tablet_bootstrap.cc:654] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:53.473486 2017 tablet_bootstrap.cc:492] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035: No bootstrap required, opened a new log
I20260620 02:58:53.473708 2017 ts_tablet_manager.cc:1397] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.009s
I20260620 02:58:53.474793 2017 raft_consensus.cc:348] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } }
I20260620 02:58:53.475106 2017 raft_consensus.cc:374] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:53.475667 2017 raft_consensus.cc:729] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d68e73fdd1142f9972b53d81b67e035, State: Initialized, Role: FOLLOWER
I20260620 02:58:53.476068 2017 consensus_queue.cc:260] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } }
I20260620 02:58:53.479743 2041 ts_tablet_manager.cc:1918] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:58:53.481426 2017 ts_tablet_manager.cc:1428] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20260620 02:58:53.482025 2040 tablet_replica.cc:330] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035: stopping tablet replica
I20260620 02:58:53.482386 2040 raft_consensus.cc:2229] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:58:53.482391 2041 log.cc:1196] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/242d966178a940cc91cc48fbcd8d98ae
I20260620 02:58:53.482733 2040 raft_consensus.cc:2258] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:58:53.483363 2041 ts_tablet_manager.cc:1939] T 242d966178a940cc91cc48fbcd8d98ae P 60260c3e799d472891e300768983b20d: Deleting consensus metadata
I20260620 02:58:53.483762 2039 tablet_replica.cc:330] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf: stopping tablet replica
I20260620 02:58:53.485754 1495 catalog_manager.cc:4841] TS 60260c3e799d472891e300768983b20d (127.0.97.2:45167): tablet 242d966178a940cc91cc48fbcd8d98ae (table test-table [id=19c7a2938528482bae60e811a3e5e928]) successfully deleted
I20260620 02:58:53.487341 2015 tablet_bootstrap.cc:492] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf: No bootstrap required, opened a new log
I20260620 02:58:53.487504 2015 ts_tablet_manager.cc:1397] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.011s
I20260620 02:58:53.488374 2040 ts_tablet_manager.cc:1905] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:58:53.488327 2015 raft_consensus.cc:348] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } }
I20260620 02:58:53.488557 2015 raft_consensus.cc:374] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:53.488658 2015 raft_consensus.cc:729] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b7439223404fe6a73b86c5a0b8abcf, State: Initialized, Role: FOLLOWER
I20260620 02:58:53.488934 2015 consensus_queue.cc:260] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } }
I20260620 02:58:53.489477 2015 ts_tablet_manager.cc:1428] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 02:58:53.489734 2039 raft_consensus.cc:2229] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:58:53.490100 2039 raft_consensus.cc:2258] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:58:53.491845 2039 ts_tablet_manager.cc:1905] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:58:53.497319 2039 ts_tablet_manager.cc:1918] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:58:53.497480 2039 log.cc:1196] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/242d966178a940cc91cc48fbcd8d98ae
I20260620 02:58:53.501513 2040 ts_tablet_manager.cc:1918] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:58:53.501714 2040 log.cc:1196] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/242d966178a940cc91cc48fbcd8d98ae
I20260620 02:58:53.502159 2039 ts_tablet_manager.cc:1939] T 242d966178a940cc91cc48fbcd8d98ae P c1b7439223404fe6a73b86c5a0b8abcf: Deleting consensus metadata
I20260620 02:58:53.502205 2040 ts_tablet_manager.cc:1939] T 242d966178a940cc91cc48fbcd8d98ae P 1d68e73fdd1142f9972b53d81b67e035: Deleting consensus metadata
I20260620 02:58:53.502877 2016 tablet_bootstrap.cc:492] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d: No bootstrap required, opened a new log
I20260620 02:58:53.503057 2016 ts_tablet_manager.cc:1397] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d: Time spent bootstrapping tablet: real 0.048s user 0.016s sys 0.006s
I20260620 02:58:53.503927 2016 raft_consensus.cc:348] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } }
I20260620 02:58:53.504176 2016 raft_consensus.cc:374] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:53.504276 2016 raft_consensus.cc:729] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60260c3e799d472891e300768983b20d, State: Initialized, Role: FOLLOWER
I20260620 02:58:53.503957 1496 catalog_manager.cc:4841] TS c1b7439223404fe6a73b86c5a0b8abcf (127.0.97.1:40209): tablet 242d966178a940cc91cc48fbcd8d98ae (table test-table [id=19c7a2938528482bae60e811a3e5e928]) successfully deleted
I20260620 02:58:53.504420 1497 catalog_manager.cc:4841] TS 1d68e73fdd1142f9972b53d81b67e035 (127.0.97.3:33023): tablet 242d966178a940cc91cc48fbcd8d98ae (table test-table [id=19c7a2938528482bae60e811a3e5e928]) successfully deleted
I20260620 02:58:53.504550 2016 consensus_queue.cc:260] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } }
I20260620 02:58:53.505208 2016 ts_tablet_manager.cc:1428] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 02:58:53.526480 2021 raft_consensus.cc:482] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:58:53.526701 2021 raft_consensus.cc:504] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } }
I20260620 02:58:53.528512 2021 leader_election.cc:288] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1b7439223404fe6a73b86c5a0b8abcf (127.0.97.1:40209), 60260c3e799d472891e300768983b20d (127.0.97.2:45167)
I20260620 02:58:53.536453 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.528818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54127 (local address 127.0.97.2:45167)
0620 02:58:53.528996 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:53.529006 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:53.530219 (+ 1213us) server_negotiation.cc:407] Connection header received
0620 02:58:53.530439 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:53.530444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:53.530521 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:53.530614 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:53.531346 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.532412 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.534221 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.535337 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.535499 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.535590 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:53.535987 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 02:58:53.536178 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:58:53.537181 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.528821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45167 (local address 127.0.97.3:54127)
0620 02:58:53.530126 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:53.530143 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:53.530304 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:53.530672 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:53.530677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:53.530704 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:53.531184 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.531195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.532531 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.532537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.534060 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.534072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.535546 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.535552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.535728 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.535773 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 02:58:53.536697 (+ 924us) client_negotiation.cc:240] Negotiation successful
0620 02:58:53.536835 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1210,"mutex_wait_us":3,"thread_start_us":1049,"threads_started":1}
I20260620 02:58:53.537520 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.529171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58533 (local address 127.0.97.1:40209)
0620 02:58:53.529376 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:53.529383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:53.529969 (+ 586us) server_negotiation.cc:407] Connection header received
0620 02:58:53.530258 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:53.530264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:53.530359 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:53.530470 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:53.531097 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.532211 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.534724 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.535937 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.536046 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.536149 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:53.537149 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0620 02:58:53.537300 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:58:53.537591 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.528687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40209 (local address 127.0.97.3:58533)
0620 02:58:53.529854 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:53.529874 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:53.530084 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:53.530488 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:53.530492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:53.530514 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:53.530920 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.530941 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.532352 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.532358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.534546 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.534560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.536332 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.536338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.536956 (+ 618us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.536994 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 02:58:53.537102 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 02:58:53.537245 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"thread_start_us":883,"threads_started":1}
I20260620 02:58:53.538372 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1185225a64b74e6bacf7e2a5dcfc8e6c" candidate_uuid: "1d68e73fdd1142f9972b53d81b67e035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" is_pre_election: true
I20260620 02:58:53.538625 1668 raft_consensus.cc:2454] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d68e73fdd1142f9972b53d81b67e035 in term 0.
I20260620 02:58:53.538484 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1185225a64b74e6bacf7e2a5dcfc8e6c" candidate_uuid: "1d68e73fdd1142f9972b53d81b67e035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60260c3e799d472891e300768983b20d" is_pre_election: true
I20260620 02:58:53.538986 1807 raft_consensus.cc:2454] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d68e73fdd1142f9972b53d81b67e035 in term 0.
I20260620 02:58:53.539099 1878 leader_election.cc:302] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d68e73fdd1142f9972b53d81b67e035, c1b7439223404fe6a73b86c5a0b8abcf; no voters:
I20260620 02:58:53.539589 2021 raft_consensus.cc:2781] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:58:53.539716 2021 raft_consensus.cc:482] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:58:53.539814 2021 raft_consensus.cc:3037] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:53.542838 2021 raft_consensus.cc:504] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } }
I20260620 02:58:53.543587 2021 leader_election.cc:288] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [CANDIDATE]: Term 1 election: Requested vote from peers c1b7439223404fe6a73b86c5a0b8abcf (127.0.97.1:40209), 60260c3e799d472891e300768983b20d (127.0.97.2:45167)
I20260620 02:58:53.544062 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1185225a64b74e6bacf7e2a5dcfc8e6c" candidate_uuid: "1d68e73fdd1142f9972b53d81b67e035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b7439223404fe6a73b86c5a0b8abcf"
I20260620 02:58:53.544251 1668 raft_consensus.cc:3037] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:53.545783 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1185225a64b74e6bacf7e2a5dcfc8e6c" candidate_uuid: "1d68e73fdd1142f9972b53d81b67e035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60260c3e799d472891e300768983b20d"
I20260620 02:58:53.545975 1807 raft_consensus.cc:3037] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:53.547154 1668 raft_consensus.cc:2454] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d68e73fdd1142f9972b53d81b67e035 in term 1.
I20260620 02:58:53.547535 1878 leader_election.cc:302] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d68e73fdd1142f9972b53d81b67e035, c1b7439223404fe6a73b86c5a0b8abcf; no voters:
I20260620 02:58:53.547821 2021 raft_consensus.cc:2781] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:58:53.548523 2021 raft_consensus.cc:686] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [term 1 LEADER]: Becoming Leader. State: Replica: 1d68e73fdd1142f9972b53d81b67e035, State: Running, Role: LEADER
I20260620 02:58:53.548942 2021 consensus_queue.cc:237] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } }
I20260620 02:58:53.549297 1807 raft_consensus.cc:2454] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d68e73fdd1142f9972b53d81b67e035 in term 1.
I20260620 02:58:53.552493 1509 catalog_manager.cc:5495] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d68e73fdd1142f9972b53d81b67e035 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "1d68e73fdd1142f9972b53d81b67e035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } health_report { overall_health: HEALTHY } } }
Jun 20 02:58:53 dist-test-slave-f1j0 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:53.601194 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.579876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.0.97.62:35375)
0620 02:58:53.581280 (+ 1404us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:53.581284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:53.581299 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:58:53.581337 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:53.581341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:53.581389 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:53.581508 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:53.582571 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.583356 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.584571 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.584970 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.585125 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.595887 (+ 10762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:53.595892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:53.595895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:53.595922 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:53.598512 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:53.599164 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:53.599171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:53.599174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:53.599295 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:53.599645 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:53.599652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:53.599655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:53.599912 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:53.600066 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:53.600456 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 02:58:53.600739 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":1285,"thread_start_us":708,"threads_started":1}
I20260620 02:58:53.601255 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.579852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35375 (local address 127.0.0.1:40764)
0620 02:58:53.580548 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:53.580571 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:53.580735 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:53.581532 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:53.581536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:53.582213 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:53.582399 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.582407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.583477 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.583485 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.584387 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.584398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.585145 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.585150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.585321 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.592871 (+ 7550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:53.592890 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:53.595683 (+ 2793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:53.598645 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:53.598659 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:53.598667 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:53.599049 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:53.599394 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:53.599401 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:53.599404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:53.599545 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:53.600040 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:53.600049 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:53.600219 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 02:58:53.600382 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 02:58:53.600846 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":189,"thread_start_us":378,"threads_started":1}
I20260620 02:58:53.607419 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40764:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 02:58:53.608436 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:58:53.615623 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 4acbb62c5d7b43009429b32e0ddeb259 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b32b882dd5be43e1b3ef3cd6755ed1e4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:58:53.615980 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 4acbb62c5d7b43009429b32e0ddeb259 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b32b882dd5be43e1b3ef3cd6755ed1e4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:58:53.616233 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4acbb62c5d7b43009429b32e0ddeb259. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:53.616179 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 4acbb62c5d7b43009429b32e0ddeb259 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b32b882dd5be43e1b3ef3cd6755ed1e4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:58:53.616467 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4acbb62c5d7b43009429b32e0ddeb259. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:53.616703 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4acbb62c5d7b43009429b32e0ddeb259. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:53.623416 2017 tablet_bootstrap.cc:492] T 4acbb62c5d7b43009429b32e0ddeb259 P 1d68e73fdd1142f9972b53d81b67e035: Bootstrap starting.
I20260620 02:58:53.624887 2016 tablet_bootstrap.cc:492] T 4acbb62c5d7b43009429b32e0ddeb259 P 60260c3e799d472891e300768983b20d: Bootstrap starting.
I20260620 02:58:53.625119 2017 tablet_bootstrap.cc:654] T 4acbb62c5d7b43009429b32e0ddeb259 P 1d68e73fdd1142f9972b53d81b67e035: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:53.625428 2015 tablet_bootstrap.cc:492] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf: Bootstrap starting.
I20260620 02:58:53.626942 2016 tablet_bootstrap.cc:654] T 4acbb62c5d7b43009429b32e0ddeb259 P 60260c3e799d472891e300768983b20d: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:53.627092 2015 tablet_bootstrap.cc:654] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:53.636827 2015 tablet_bootstrap.cc:492] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf: No bootstrap required, opened a new log
I20260620 02:58:53.637064 2015 ts_tablet_manager.cc:1397] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260620 02:58:53.637851 2015 raft_consensus.cc:348] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } }
I20260620 02:58:53.638180 2015 raft_consensus.cc:374] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:53.638294 2015 raft_consensus.cc:729] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b7439223404fe6a73b86c5a0b8abcf, State: Initialized, Role: FOLLOWER
I20260620 02:58:53.638607 2015 consensus_queue.cc:260] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } }
I20260620 02:58:53.639133 2022 tablet_replica.cc:439] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } } }
I20260620 02:58:53.639473 2022 tablet_replica.cc:442] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:58:53.639573 2016 tablet_bootstrap.cc:492] T 4acbb62c5d7b43009429b32e0ddeb259 P 60260c3e799d472891e300768983b20d: No bootstrap required, opened a new log
I20260620 02:58:53.639744 2016 ts_tablet_manager.cc:1397] T 4acbb62c5d7b43009429b32e0ddeb259 P 60260c3e799d472891e300768983b20d: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.010s
I20260620 02:58:53.640165 2017 tablet_bootstrap.cc:492] T 4acbb62c5d7b43009429b32e0ddeb259 P 1d68e73fdd1142f9972b53d81b67e035: No bootstrap required, opened a new log
I20260620 02:58:53.640331 2017 ts_tablet_manager.cc:1397] T 4acbb62c5d7b43009429b32e0ddeb259 P 1d68e73fdd1142f9972b53d81b67e035: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.011s
I20260620 02:58:53.640503 2016 raft_consensus.cc:348] T 4acbb62c5d7b43009429b32e0ddeb259 P 60260c3e799d472891e300768983b20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } }
I20260620 02:58:53.640749 2016 raft_consensus.cc:374] T 4acbb62c5d7b43009429b32e0ddeb259 P 60260c3e799d472891e300768983b20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:53.640846 2016 raft_consensus.cc:729] T 4acbb62c5d7b43009429b32e0ddeb259 P 60260c3e799d472891e300768983b20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60260c3e799d472891e300768983b20d, State: Initialized, Role: FOLLOWER
I20260620 02:58:53.641055 2017 raft_consensus.cc:348] T 4acbb62c5d7b43009429b32e0ddeb259 P 1d68e73fdd1142f9972b53d81b67e035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } }
I20260620 02:58:53.641300 2017 raft_consensus.cc:374] T 4acbb62c5d7b43009429b32e0ddeb259 P 1d68e73fdd1142f9972b53d81b67e035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:53.641386 2017 raft_consensus.cc:729] T 4acbb62c5d7b43009429b32e0ddeb259 P 1d68e73fdd1142f9972b53d81b67e035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d68e73fdd1142f9972b53d81b67e035, State: Initialized, Role: FOLLOWER
I20260620 02:58:53.641674 2017 consensus_queue.cc:260] T 4acbb62c5d7b43009429b32e0ddeb259 P 1d68e73fdd1142f9972b53d81b67e035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } }
I20260620 02:58:53.642045 2016 consensus_queue.cc:260] T 4acbb62c5d7b43009429b32e0ddeb259 P 60260c3e799d472891e300768983b20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } }
I20260620 02:58:53.642093 2021 tablet_replica.cc:439] T 4acbb62c5d7b43009429b32e0ddeb259 P 1d68e73fdd1142f9972b53d81b67e035: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } } }
I20260620 02:58:53.642383 2021 tablet_replica.cc:442] T 4acbb62c5d7b43009429b32e0ddeb259 P 1d68e73fdd1142f9972b53d81b67e035: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:58:53.642648 2015 ts_tablet_manager.cc:1428] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 02:58:53.642494 2034 tablet_replica.cc:439] T 4acbb62c5d7b43009429b32e0ddeb259 P 60260c3e799d472891e300768983b20d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } } }
I20260620 02:58:53.642858 2034 tablet_replica.cc:442] T 4acbb62c5d7b43009429b32e0ddeb259 P 60260c3e799d472891e300768983b20d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:58:53.643774 2016 ts_tablet_manager.cc:1428] T 4acbb62c5d7b43009429b32e0ddeb259 P 60260c3e799d472891e300768983b20d: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260620 02:58:53.645704 2017 ts_tablet_manager.cc:1428] T 4acbb62c5d7b43009429b32e0ddeb259 P 1d68e73fdd1142f9972b53d81b67e035: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260620 02:58:53.733341 2022 raft_consensus.cc:482] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:58:53.733626 2022 raft_consensus.cc:504] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } }
I20260620 02:58:53.735464 2022 leader_election.cc:288] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60260c3e799d472891e300768983b20d (127.0.97.2:45167), 1d68e73fdd1142f9972b53d81b67e035 (127.0.97.3:33023)
I20260620 02:58:53.744360 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.736158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32919 (local address 127.0.97.2:45167)
0620 02:58:53.736345 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:53.736350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:53.737077 (+ 727us) server_negotiation.cc:407] Connection header received
0620 02:58:53.737225 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:53.737231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:53.737332 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:53.737444 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:53.738102 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.738829 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.742147 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.743469 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.743614 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.743715 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:53.743951 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 02:58:53.744080 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:58:53.744431 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.735765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45167 (local address 127.0.97.1:32919)
0620 02:58:53.736952 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:53.736973 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:53.737089 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:53.737471 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:53.737476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:53.737501 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:53.737963 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.737972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.738955 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.738961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.741986 (+ 3025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.742000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.743664 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.743668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.743790 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.743813 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 02:58:53.743925 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 02:58:53.744052 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"mutex_wait_us":190,"thread_start_us":823,"threads_started":1}
I20260620 02:58:53.745132 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4acbb62c5d7b43009429b32e0ddeb259" candidate_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60260c3e799d472891e300768983b20d" is_pre_election: true
I20260620 02:58:53.745434 1807 raft_consensus.cc:2454] T 4acbb62c5d7b43009429b32e0ddeb259 P 60260c3e799d472891e300768983b20d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1b7439223404fe6a73b86c5a0b8abcf in term 0.
I20260620 02:58:53.746068 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.735914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60293 (local address 127.0.97.3:33023)
0620 02:58:53.736112 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:53.736118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:53.736673 (+ 555us) server_negotiation.cc:407] Connection header received
0620 02:58:53.736975 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:53.736982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:53.737064 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:53.737151 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:53.737874 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.739037 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.743802 (+ 4765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.744991 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.745148 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.745243 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:53.745643 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 02:58:53.745788 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:58:53.746078 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.735753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33023 (local address 127.0.97.1:60293)
0620 02:58:53.736558 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:53.736582 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:53.736801 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:53.737241 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:53.737246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:53.737268 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:53.737664 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.737681 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.739524 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.739529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.741985 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.742000 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.745209 (+ 3209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.745215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.745374 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.745415 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 02:58:53.745510 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 02:58:53.745671 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":131,"thread_start_us":513,"threads_started":1}
I20260620 02:58:53.746037 1598 leader_election.cc:302] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60260c3e799d472891e300768983b20d, c1b7439223404fe6a73b86c5a0b8abcf; no voters:
I20260620 02:58:53.746466 2022 raft_consensus.cc:2781] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:58:53.746575 2022 raft_consensus.cc:482] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:58:53.746659 2022 raft_consensus.cc:3037] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:53.747661 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4acbb62c5d7b43009429b32e0ddeb259" candidate_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d68e73fdd1142f9972b53d81b67e035" is_pre_election: true
I20260620 02:58:53.747912 1946 raft_consensus.cc:2454] T 4acbb62c5d7b43009429b32e0ddeb259 P 1d68e73fdd1142f9972b53d81b67e035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1b7439223404fe6a73b86c5a0b8abcf in term 0.
I20260620 02:58:53.749330 2022 raft_consensus.cc:504] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } }
I20260620 02:58:53.749950 2022 leader_election.cc:288] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf [CANDIDATE]: Term 1 election: Requested vote from peers 60260c3e799d472891e300768983b20d (127.0.97.2:45167), 1d68e73fdd1142f9972b53d81b67e035 (127.0.97.3:33023)
I20260620 02:58:53.750382 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4acbb62c5d7b43009429b32e0ddeb259" candidate_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60260c3e799d472891e300768983b20d"
I20260620 02:58:53.750452 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4acbb62c5d7b43009429b32e0ddeb259" candidate_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d68e73fdd1142f9972b53d81b67e035"
I20260620 02:58:53.750622 1807 raft_consensus.cc:3037] T 4acbb62c5d7b43009429b32e0ddeb259 P 60260c3e799d472891e300768983b20d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:53.750625 1946 raft_consensus.cc:3037] T 4acbb62c5d7b43009429b32e0ddeb259 P 1d68e73fdd1142f9972b53d81b67e035 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:53.753599 1946 raft_consensus.cc:2454] T 4acbb62c5d7b43009429b32e0ddeb259 P 1d68e73fdd1142f9972b53d81b67e035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1b7439223404fe6a73b86c5a0b8abcf in term 1.
I20260620 02:58:53.753636 1807 raft_consensus.cc:2454] T 4acbb62c5d7b43009429b32e0ddeb259 P 60260c3e799d472891e300768983b20d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1b7439223404fe6a73b86c5a0b8abcf in term 1.
I20260620 02:58:53.753993 1600 leader_election.cc:302] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d68e73fdd1142f9972b53d81b67e035, c1b7439223404fe6a73b86c5a0b8abcf; no voters:
I20260620 02:58:53.754763 2022 raft_consensus.cc:2781] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:58:53.755276 2022 raft_consensus.cc:686] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf [term 1 LEADER]: Becoming Leader. State: Replica: c1b7439223404fe6a73b86c5a0b8abcf, State: Running, Role: LEADER
I20260620 02:58:53.755858 2022 consensus_queue.cc:237] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } }
I20260620 02:58:53.757855 2061 tablet_replica.cc:439] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf: TxnStatusTablet state changed. Reason: New leader c1b7439223404fe6a73b86c5a0b8abcf. Latest consensus state: current_term: 1 leader_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } } }
I20260620 02:58:53.758267 2061 tablet_replica.cc:442] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:58:53.758950 2063 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 02:58:53.759224 2063 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 02:58:53.759848 1511 catalog_manager.cc:5495] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf reported cstate change: term changed from 0 to 1, leader changed from <none> to c1b7439223404fe6a73b86c5a0b8abcf (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } health_report { overall_health: UNKNOWN } } }
I20260620 02:58:53.796373 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.786004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40209 (local address 127.0.0.1:36382)
0620 02:58:53.786419 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:53.786441 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:53.786752 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:53.787728 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:53.787851 (+ 123us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:53.787869 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:58:53.788886 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.788904 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.791060 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.791066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.792742 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.792759 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.794175 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.794181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.794418 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.794501 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:58:53.795781 (+ 1280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:58:53.795800 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 02:58:53.795885 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 02:58:53.796052 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 02:58:53.797030 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.785455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36382 (local address 127.0.97.1:40209)
0620 02:58:53.785769 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:53.785779 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:53.786549 (+ 770us) server_negotiation.cc:407] Connection header received
0620 02:58:53.787101 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:53.787107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:53.787339 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:53.787789 (+ 450us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:58:53.789107 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.790835 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.792976 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.794014 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.794163 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.794758 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:58:53.795575 (+ 817us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:58:53.795727 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:53.796699 (+ 972us) server_negotiation.cc:299] Negotiation successful
0620 02:58:53.796783 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 02:58:53.800923 1946 raft_consensus.cc:1261] T 4acbb62c5d7b43009429b32e0ddeb259 P 1d68e73fdd1142f9972b53d81b67e035 [term 1 FOLLOWER]: Refusing update from remote peer c1b7439223404fe6a73b86c5a0b8abcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:58:53.801378 1807 raft_consensus.cc:1261] T 4acbb62c5d7b43009429b32e0ddeb259 P 60260c3e799d472891e300768983b20d [term 1 FOLLOWER]: Refusing update from remote peer c1b7439223404fe6a73b86c5a0b8abcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:58:53.801673 2022 consensus_queue.cc:1035] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:58:53.802258 2022 consensus_queue.cc:1035] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:58:53.816071 2021 tablet_replica.cc:439] T 4acbb62c5d7b43009429b32e0ddeb259 P 1d68e73fdd1142f9972b53d81b67e035: TxnStatusTablet state changed. Reason: New leader c1b7439223404fe6a73b86c5a0b8abcf. Latest consensus state: current_term: 1 leader_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } } }
I20260620 02:58:53.816373 2021 tablet_replica.cc:442] T 4acbb62c5d7b43009429b32e0ddeb259 P 1d68e73fdd1142f9972b53d81b67e035: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:58:53.816845 2034 tablet_replica.cc:439] T 4acbb62c5d7b43009429b32e0ddeb259 P 60260c3e799d472891e300768983b20d: TxnStatusTablet state changed. Reason: New leader c1b7439223404fe6a73b86c5a0b8abcf. Latest consensus state: current_term: 1 leader_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } } }
I20260620 02:58:53.817121 2034 tablet_replica.cc:442] T 4acbb62c5d7b43009429b32e0ddeb259 P 60260c3e799d472891e300768983b20d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:58:53.818567 2061 tablet_replica.cc:439] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } } }
I20260620 02:58:53.818813 2061 tablet_replica.cc:442] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:58:53.820091 2031 mvcc.cc:205] Tried to move back new op lower bound from 7298762071241850880 to 7298762071067508736. Current Snapshot: MvccSnapshot[applied={T|T < 7298762071241850880}]
I20260620 02:58:53.821653 2021 tablet_replica.cc:439] T 4acbb62c5d7b43009429b32e0ddeb259 P 1d68e73fdd1142f9972b53d81b67e035: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } } }
I20260620 02:58:53.821712 2061 tablet_replica.cc:439] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } } }
I20260620 02:58:53.821908 2061 tablet_replica.cc:442] T 4acbb62c5d7b43009429b32e0ddeb259 P c1b7439223404fe6a73b86c5a0b8abcf: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:58:53.821909 2021 tablet_replica.cc:442] T 4acbb62c5d7b43009429b32e0ddeb259 P 1d68e73fdd1142f9972b53d81b67e035: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:58:53.824077 2034 tablet_replica.cc:439] T 4acbb62c5d7b43009429b32e0ddeb259 P 60260c3e799d472891e300768983b20d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 } } peers { permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 } } peers { permanent_uuid: "1d68e73fdd1142f9972b53d81b67e035" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33023 } } }
I20260620 02:58:53.824307 2034 tablet_replica.cc:442] T 4acbb62c5d7b43009429b32e0ddeb259 P 60260c3e799d472891e300768983b20d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:58:53.839044 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.830516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59246 (local address 127.0.97.3:33023)
0620 02:58:53.830735 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:53.830743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:53.830765 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:58:53.830879 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:53.830887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:53.831007 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:53.831185 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:58:53.831873 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.833328 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.836516 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.837042 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.837191 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.837770 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:58:53.838332 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:58:53.838503 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:53.838592 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 02:58:53.838672 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:58:53.855408 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.849178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38343 (local address 127.0.97.1:40209)
0620 02:58:53.849417 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:53.849425 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:53.850163 (+ 738us) server_negotiation.cc:407] Connection header received
0620 02:58:53.850376 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:53.850384 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:53.850452 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:53.850541 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:58:53.851021 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.851723 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.853073 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.854012 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.854176 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.854599 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:58:53.854694 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:58:53.854790 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:53.854942 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 02:58:53.855015 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 02:58:53.855408 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.849084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40209 (local address 127.0.97.3:38343)
0620 02:58:53.850047 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:53.850067 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:53.850233 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:53.850568 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:53.850573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:53.850585 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:58:53.850875 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.850883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.851846 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.851854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.852924 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.852935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.854216 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.854225 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.854363 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.854415 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:58:53.854789 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:58:53.854802 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 02:58:53.854903 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 02:58:53.855045 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":695,"threads_started":1}
I20260620 02:58:53.865968 1806 raft_consensus.cc:1261] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d [term 1 FOLLOWER]: Refusing update from remote peer 1d68e73fdd1142f9972b53d81b67e035: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:58:53.866842 2047 consensus_queue.cc:1035] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60260c3e799d472891e300768983b20d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:58:53.867338 1668 raft_consensus.cc:1261] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf [term 1 FOLLOWER]: Refusing update from remote peer 1d68e73fdd1142f9972b53d81b67e035: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:58:53.867898 2047 consensus_queue.cc:1035] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b7439223404fe6a73b86c5a0b8abcf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:58:53.882834 2029 mvcc.cc:205] Tried to move back new op lower bound from 7298762071507922944 to 7298762070221094912. Current Snapshot: MvccSnapshot[applied={T|T < 7298762071507922944}]
I20260620 02:58:53.935797 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.915731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38235 (local address 127.0.97.3:33023)
0620 02:58:53.915888 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:53.915894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:53.928187 (+ 12293us) server_negotiation.cc:407] Connection header received
0620 02:58:53.928729 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:53.928734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:53.928828 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:53.928895 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:58:53.929755 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.930863 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.933039 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.934202 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.934314 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.935013 (+ 699us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:58:53.935143 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:58:53.935209 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:53.935415 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 02:58:53.935509 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 02:58:53.936602 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:53.915352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33023 (local address 127.0.97.1:38235)
0620 02:58:53.928076 (+ 12724us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:53.928104 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:53.928549 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:53.929136 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:53.929141 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:53.929152 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:58:53.929542 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.929557 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.931064 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.931070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.932826 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:53.932839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:53.934540 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:53.934546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:53.934706 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:53.934784 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:58:53.935289 (+ 505us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:58:53.935297 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 02:58:53.936069 (+ 772us) client_negotiation.cc:240] Negotiation successful
0620 02:58:53.936213 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12457,"thread_start_us":12858,"threads_started":1}
I20260620 02:58:54.030551 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:58:54.030802 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:58:54.034438 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbbcafcba8034a5a84449a352bfa8550: Sending DeleteTablet for 3 replicas of tablet 1185225a64b74e6bacf7e2a5dcfc8e6c
I20260620 02:58:54.035624 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 1185225a64b74e6bacf7e2a5dcfc8e6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:58:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58502
I20260620 02:58:54.035622 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 1185225a64b74e6bacf7e2a5dcfc8e6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:58:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57498
I20260620 02:58:54.035660 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 1185225a64b74e6bacf7e2a5dcfc8e6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:58:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58192
I20260620 02:58:54.058130 2082 tablet_replica.cc:330] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf: stopping tablet replica
I20260620 02:58:54.058444 2082 raft_consensus.cc:2229] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:58:54.058758 2082 raft_consensus.cc:2258] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:58:54.064088 2082 ts_tablet_manager.cc:1905] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:58:54.070648 2080 tablet_replica.cc:330] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035: stopping tablet replica
I20260620 02:58:54.070881 2080 raft_consensus.cc:2229] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:58:54.071442 2080 raft_consensus.cc:2258] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:58:54.074366 2080 ts_tablet_manager.cc:1905] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:58:54.081218 2081 tablet_replica.cc:330] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d: stopping tablet replica
I20260620 02:58:54.081424 2081 raft_consensus.cc:2229] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:58:54.081444 2080 ts_tablet_manager.cc:1918] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:58:54.081668 2080 log.cc:1196] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/1185225a64b74e6bacf7e2a5dcfc8e6c
I20260620 02:58:54.081673 2081 raft_consensus.cc:2258] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:58:54.082125 2080 ts_tablet_manager.cc:1939] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 1d68e73fdd1142f9972b53d81b67e035: Deleting consensus metadata
I20260620 02:58:54.083242 1497 catalog_manager.cc:4841] TS 1d68e73fdd1142f9972b53d81b67e035 (127.0.97.3:33023): tablet 1185225a64b74e6bacf7e2a5dcfc8e6c (table test-table [id=2b04294be61547bfb893a7c64553d5e6]) successfully deleted
I20260620 02:58:54.084554 2081 ts_tablet_manager.cc:1905] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:58:54.089490 2081 ts_tablet_manager.cc:1918] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:58:54.089653 2081 log.cc:1196] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/1185225a64b74e6bacf7e2a5dcfc8e6c
I20260620 02:58:54.090128 2081 ts_tablet_manager.cc:1939] T 1185225a64b74e6bacf7e2a5dcfc8e6c P 60260c3e799d472891e300768983b20d: Deleting consensus metadata
I20260620 02:58:54.091343 1495 catalog_manager.cc:4841] TS 60260c3e799d472891e300768983b20d (127.0.97.2:45167): tablet 1185225a64b74e6bacf7e2a5dcfc8e6c (table test-table [id=2b04294be61547bfb893a7c64553d5e6]) successfully deleted
I20260620 02:58:54.092163 2082 ts_tablet_manager.cc:1918] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:58:54.092324 2082 log.cc:1196] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/1185225a64b74e6bacf7e2a5dcfc8e6c
I20260620 02:58:54.092826 2082 ts_tablet_manager.cc:1939] T 1185225a64b74e6bacf7e2a5dcfc8e6c P c1b7439223404fe6a73b86c5a0b8abcf: Deleting consensus metadata
Jun 20 02:58:54 dist-test-slave-f1j0 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924332, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:58:54.093967 1496 catalog_manager.cc:4841] TS c1b7439223404fe6a73b86c5a0b8abcf (127.0.97.1:40209): tablet 1185225a64b74e6bacf7e2a5dcfc8e6c (table test-table [id=2b04294be61547bfb893a7c64553d5e6]) successfully deleted
I20260620 02:58:54.101280 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:54.060773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36392 (local address 127.0.97.1:40209)
0620 02:58:54.061008 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:54.061017 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:54.065312 (+ 4295us) server_negotiation.cc:407] Connection header received
0620 02:58:54.066097 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:54.066106 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:54.066218 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:54.066309 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:54.077681 (+ 11372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:54.078992 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:54.081913 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:54.082582 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:54.082694 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:54.094660 (+ 11966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:54.094673 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:54.094691 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:54.094733 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:54.098304 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:54.099076 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:54.099084 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:54.099086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:54.099213 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:54.099594 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:54.099605 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:54.099608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:54.099949 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:54.100297 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:54.100521 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 02:58:54.100727 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 02:58:54.102206 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36392
I20260620 02:58:54.133450 2087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:54.105256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.0.97.62:35375)
0620 02:58:54.119637 (+ 14381us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:54.119641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:54.119660 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:58:54.119720 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:54.119723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:54.119789 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:54.119934 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:54.121544 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:54.122346 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:54.123609 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:54.124049 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:54.124183 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:54.126710 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:54.126716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:54.126722 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:54.126772 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:54.130362 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:54.131062 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:54.131069 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:54.131075 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:54.131237 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:54.131717 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:54.131724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:54.131730 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:54.132125 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:54.132338 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:54.132789 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 02:58:54.133006 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14246,"thread_start_us":14136,"threads_started":1}
W20260620 02:58:54.134289 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40776
I20260620 02:58:54.136793 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 1582
I20260620 02:58:54.152783 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:54.137813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.0.97.1:40209)
0620 02:58:54.138146 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:54.138152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:54.140002 (+ 1850us) server_negotiation.cc:407] Connection header received
0620 02:58:54.140197 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:54.140202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:54.140277 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:54.140381 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:54.141451 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:54.142309 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:54.143474 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:54.144339 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:54.144465 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:54.147119 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:54.147125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:54.147136 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:54.147168 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:54.150064 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:54.150707 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:54.150718 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:54.150721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:54.150842 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:54.151222 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:54.151228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:54.151232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:54.151500 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:54.151803 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:54.152080 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 02:58:54.152309 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
W20260620 02:58:54.153379 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36400
W20260620 02:58:54.153930 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 1582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:58:54.154127 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 1582
I20260620 02:58:54.174749 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 1722
Jun 20 02:58:54 dist-test-slave-f1j0 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924332, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:58:54.191004 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:54.175554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42354 (local address 127.0.97.2:45167)
0620 02:58:54.175738 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:54.175744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:54.175765 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:58:54.175908 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:54.175914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:54.175994 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:54.176116 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:54.177284 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:54.178413 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:54.179677 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:54.180294 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:54.180414 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:54.184520 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:54.184538 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:54.184560 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:54.184598 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:54.188294 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:54.188874 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:54.188884 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:54.188890 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:54.189045 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:54.189387 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:54.189392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:54.189412 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:54.189847 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:54.190175 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:54.190369 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 02:58:54.190623 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 02:58:54.191844 1852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42354
W20260620 02:58:54.192534 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 1722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:58:54.192677 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 1722
I20260620 02:58:54.214046 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 1861
Jun 20 02:58:54 dist-test-slave-f1j0 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924332, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:58:54.230623 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:54.214795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.0.97.3:33023)
0620 02:58:54.214999 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:54.215005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:54.215077 (+ 72us) server_negotiation.cc:407] Connection header received
0620 02:58:54.215293 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:54.215297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:54.215359 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:54.215480 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:54.216816 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:54.217691 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:54.218897 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:54.219316 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:54.219445 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:54.223758 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:54.223780 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:54.223802 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:54.223859 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:54.227813 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:54.228434 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:54.228443 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:54.228447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:54.228615 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:54.228940 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:54.228946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:54.228949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:54.229390 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:54.229741 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:54.230028 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 02:58:54.230247 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260620 02:58:54.231391 1991 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59256
W20260620 02:58:54.232023 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 1861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:58:54.232194 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 1861
I20260620 02:58:54.250675 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 1479
I20260620 02:58:54.251816 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:58:54.364413 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 1479
2026-06-20T02:58:54Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4715 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:58:54 dist-test-slave-f1j0 krb5kdc[2092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:58:54 dist-test-slave-f1j0 krb5kdc[2092](info): set up 2 sockets
Jun 20 02:58:54 dist-test-slave-f1j0 krb5kdc[2092](info): commencing operation
krb5kdc: starting...
W20260620 02:58:55.437817 388 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 02:58:55 dist-test-slave-f1j0 krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:58:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:58:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:58:55.622469 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40285
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:58:55.932492 2105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:58:55.932777 2105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:58:55.932866 2105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:58:55.940622 2105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:58:55.940737 2105 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 02:58:55.940778 2105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:58:55.940829 2105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:58:55.940884 2105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:58:55.940946 2105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:58:55.950922 2105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40285
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:58:55.952186 2105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:58:55.954020 2105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:58:55.965981 2110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:55.966041 2111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:55.966012 2113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:58:55.966866 2105 server_base.cc:1034] running on GCE node
I20260620 02:58:55.967679 2105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:58:55.969045 2105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:58:55.970279 2105 hybrid_clock.cc:648] HybridClock initialized: now 1781924335970219 us; error 41 us; skew 500 ppm
I20260620 02:58:55.970605 2105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:58:55 dist-test-slave-f1j0 krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:58:55.973635 2105 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:58:55.975014 2105 webserver.cc:460] Webserver started at http://127.0.97.62:42663/ using document root <none> and password file <none>
I20260620 02:58:55.975704 2105 fs_manager.cc:359] Metadata directory not provided
I20260620 02:58:55.975839 2105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:58:55.976148 2105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:58:55.979075 2105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "2db0c28a722c4a6fbf20e9eb269bf045"
format_stamp: "Formatted at 2026-06-20 02:58:55 on dist-test-slave-f1j0"
server_key: "1ceafb1ba7e21c960460f0d54b6fda3c"
server_key_iv: "541a7a20541866bbd66bf283f2615513"
server_key_version: "encryptionkey@0"
I20260620 02:58:55.979830 2105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "2db0c28a722c4a6fbf20e9eb269bf045"
format_stamp: "Formatted at 2026-06-20 02:58:55 on dist-test-slave-f1j0"
server_key: "1ceafb1ba7e21c960460f0d54b6fda3c"
server_key_iv: "541a7a20541866bbd66bf283f2615513"
server_key_version: "encryptionkey@0"
I20260620 02:58:55.986280 2105 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260620 02:58:55.989742 2120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:55.991057 2105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:58:55.991243 2105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "2db0c28a722c4a6fbf20e9eb269bf045"
format_stamp: "Formatted at 2026-06-20 02:58:55 on dist-test-slave-f1j0"
server_key: "1ceafb1ba7e21c960460f0d54b6fda3c"
server_key_iv: "541a7a20541866bbd66bf283f2615513"
server_key_version: "encryptionkey@0"
I20260620 02:58:55.991457 2105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:58:56.006258 2105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:58:56.010281 2105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:58:56.010541 2105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:58:56.031430 2105 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40285
I20260620 02:58:56.031450 2181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40285 every 8 connection(s)
I20260620 02:58:56.032858 2105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:58:56.036566 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:56.039902 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 2105
I20260620 02:58:56.040143 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 02:58:56.040930 388 external_mini_cluster.cc:1428] Setting key 36c0d1318dc836bc2e4adaff6145f016
I20260620 02:58:56.048722 2182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045: Bootstrap starting.
I20260620 02:58:56.051766 2182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:56.052735 2182 log.cc:824] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:58:56 dist-test-slave-f1j0 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:56.055698 2182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045: No bootstrap required, opened a new log
I20260620 02:58:56.063685 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:56.043047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41238 (local address 127.0.97.62:40285)
0620 02:58:56.043756 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:56.043768 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:56.043807 (+ 39us) server_negotiation.cc:407] Connection header received
0620 02:58:56.045440 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:56.045461 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:56.045930 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:56.046339 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:56.047723 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:56.049031 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:56.050289 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:56.050926 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:56.051109 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:56.055169 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:56.055187 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:56.055206 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:56.055252 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:56.059235 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:56.059898 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:56.059907 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:56.059915 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:56.060115 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:56.060504 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:56.060510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:56.060513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:56.061169 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:56.061469 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:56.061899 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 02:58:56.062342 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":312,"threads_started":1}
I20260620 02:58:56.066269 2182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2db0c28a722c4a6fbf20e9eb269bf045" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40285 } }
I20260620 02:58:56.066735 2182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:56.066831 2182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db0c28a722c4a6fbf20e9eb269bf045, State: Initialized, Role: FOLLOWER
I20260620 02:58:56.067358 2182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2db0c28a722c4a6fbf20e9eb269bf045" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40285 } }
I20260620 02:58:56.067581 2182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:58:56.067713 2182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:58:56.067874 2182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:56.070288 2182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2db0c28a722c4a6fbf20e9eb269bf045" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40285 } }
I20260620 02:58:56.070952 2182 leader_election.cc:302] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2db0c28a722c4a6fbf20e9eb269bf045; no voters:
I20260620 02:58:56.071697 2182 leader_election.cc:288] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:58:56.071785 2187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:58:56.072525 2187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045 [term 1 LEADER]: Becoming Leader. State: Replica: 2db0c28a722c4a6fbf20e9eb269bf045, State: Running, Role: LEADER
I20260620 02:58:56.073153 2187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2db0c28a722c4a6fbf20e9eb269bf045" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40285 } }
I20260620 02:58:56.073887 2182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:58:56.076423 2189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2db0c28a722c4a6fbf20e9eb269bf045. Latest consensus state: current_term: 1 leader_uuid: "2db0c28a722c4a6fbf20e9eb269bf045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2db0c28a722c4a6fbf20e9eb269bf045" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40285 } } }
I20260620 02:58:56.076362 2188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2db0c28a722c4a6fbf20e9eb269bf045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2db0c28a722c4a6fbf20e9eb269bf045" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40285 } } }
I20260620 02:58:56.076759 2188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045 [sys.catalog]: This master's current role is: LEADER
I20260620 02:58:56.076759 2189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045 [sys.catalog]: This master's current role is: LEADER
I20260620 02:58:56.077556 2195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:58:56.084546 2195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:58:56.092918 2195 catalog_manager.cc:1347] Generated new cluster ID: e6898ccbaeae4644b94a7f418aea4fac
I20260620 02:58:56.093039 2195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:58:56.105136 2195 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:58:56.106767 2195 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:58:56.113653 2195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045: Generated new TSK 0
I20260620 02:58:56.114670 2195 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:58:56.160791 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40285
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:58:56.482349 2208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:58:56.482664 2208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:58:56.482831 2208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:58:56.490370 2208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:58:56.490511 2208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:58:56.490645 2208 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 02:58:56.490729 2208 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:58:56.490852 2208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:58:56.500634 2208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:58:56.501976 2208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:58:56.503309 2208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:58:56.516388 2213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:56.516358 2214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:56.516321 2216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:58:56.517068 2208 server_base.cc:1034] running on GCE node
I20260620 02:58:56.517746 2208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:58:56.518676 2208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:58:56.519866 2208 hybrid_clock.cc:648] HybridClock initialized: now 1781924336519831 us; error 34 us; skew 500 ppm
I20260620 02:58:56.520197 2208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:58:56 dist-test-slave-f1j0 krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:58:56.523329 2208 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:58:56.524653 2208 webserver.cc:460] Webserver started at http://127.0.97.1:42761/ using document root <none> and password file <none>
I20260620 02:58:56.525286 2208 fs_manager.cc:359] Metadata directory not provided
I20260620 02:58:56.525382 2208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:58:56.525694 2208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:58:56.528594 2208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "561036292d05476c840ea0e9636a1bf4"
format_stamp: "Formatted at 2026-06-20 02:58:56 on dist-test-slave-f1j0"
server_key: "d5dff3d069715e49157acb8482700331"
server_key_iv: "5a6807e16c480578bac40fdc02c67675"
server_key_version: "encryptionkey@0"
I20260620 02:58:56.529307 2208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "561036292d05476c840ea0e9636a1bf4"
format_stamp: "Formatted at 2026-06-20 02:58:56 on dist-test-slave-f1j0"
server_key: "d5dff3d069715e49157acb8482700331"
server_key_iv: "5a6807e16c480578bac40fdc02c67675"
server_key_version: "encryptionkey@0"
I20260620 02:58:56.534196 2208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 02:58:56.537402 2223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:56.538794 2208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:58:56.538954 2208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "561036292d05476c840ea0e9636a1bf4"
format_stamp: "Formatted at 2026-06-20 02:58:56 on dist-test-slave-f1j0"
server_key: "d5dff3d069715e49157acb8482700331"
server_key_iv: "5a6807e16c480578bac40fdc02c67675"
server_key_version: "encryptionkey@0"
I20260620 02:58:56.539144 2208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:58:56.555521 2208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:58:56.559782 2208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:58:56.560055 2208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:58:56.562839 2208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:58:56.562958 2208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:56.563107 2208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:58:56.563227 2208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 20 02:58:56 dist-test-slave-f1j0 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:56.586222 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:56.563267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59569 (local address 127.0.97.62:40285)
0620 02:58:56.563431 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:56.563442 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:56.564876 (+ 1434us) server_negotiation.cc:407] Connection header received
0620 02:58:56.566986 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:56.566993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:56.567084 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:56.567197 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:56.569416 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:56.570461 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:56.573056 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:56.573729 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:56.573879 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:56.579771 (+ 5892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:56.579776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:56.579779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:56.579811 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:56.582576 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:56.583714 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:56.583723 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:56.583726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:56.583880 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:56.584490 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:56.584502 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:56.584504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:56.584769 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:56.584919 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:56.585662 (+ 743us) server_negotiation.cc:299] Negotiation successful
0620 02:58:56.585889 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 02:58:56.587391 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:56.563959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40285 (local address 127.0.97.1:59569)
0620 02:58:56.564719 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:56.564760 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:56.566713 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:56.567419 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:56.567428 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:56.568172 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:56.569171 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:56.569197 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:56.570611 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:56.570622 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:56.572901 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:56.572912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:56.573992 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:56.573999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:56.574359 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:56.575245 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:56.575284 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:56.579497 (+ 4213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:56.582744 (+ 3247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:56.582753 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:56.582776 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:56.583557 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:56.584005 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:56.584011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:56.584015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:56.584323 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:56.584912 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:56.584920 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:56.585290 (+ 370us) client_negotiation.cc:769] Sending connection context
0620 02:58:56.585562 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 02:58:56.586148 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":345,"threads_started":1}
I20260620 02:58:56.607920 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:56.594018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33345 (local address 127.0.97.62:40285)
0620 02:58:56.594234 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:56.594243 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:56.595137 (+ 894us) server_negotiation.cc:407] Connection header received
0620 02:58:56.595324 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:56.595331 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:56.595434 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:56.595564 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:56.596516 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:56.597942 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:56.599952 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:56.600370 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:56.600497 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:56.602641 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:56.602646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:56.602649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:56.602676 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:56.605088 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:56.605868 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:56.605876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:56.605882 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:56.606058 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:56.606540 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:56.606545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:56.606549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:56.606841 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:56.606954 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:56.607481 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 02:58:56.607641 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:58:56.608501 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:56.594340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40285 (local address 127.0.97.1:33345)
0620 02:58:56.595045 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:56.595060 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:56.595191 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:56.595575 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:56.595578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:56.596120 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:56.596381 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:56.596390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:56.598112 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:56.598118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:56.599783 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:56.599794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:56.600553 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:56.600561 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:56.600727 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:56.601413 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:56.601428 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:56.602459 (+ 1031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:56.605259 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:56.605264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:56.605268 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:56.605725 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:56.606193 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:56.606199 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:56.606204 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:56.606406 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:56.606962 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:56.606970 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:56.607227 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 02:58:56.607421 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 02:58:56.607880 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":364,"threads_started":1}
I20260620 02:58:56.622402 2208 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38291
I20260620 02:58:56.622437 2342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38291 every 8 connection(s)
I20260620 02:58:56.623654 2208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:58:56.625705 2343 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40285
I20260620 02:58:56.625926 2343 heartbeater.cc:464] Registering TS with master...
I20260620 02:58:56.626546 2343 heartbeater.cc:511] Master 127.0.97.62:40285 requested a full tablet report, sending...
I20260620 02:58:56.628527 2137 ts_manager.cc:194] Registered new tserver with Master: 561036292d05476c840ea0e9636a1bf4 (127.0.97.1:38291)
I20260620 02:58:56.630044 2137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59569
I20260620 02:58:56.630343 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 2208
I20260620 02:58:56.630540 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:58:56.631141 388 external_mini_cluster.cc:1428] Setting key fff5d9fa435b74633f50e1aea85a291b
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:58:56.670138 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40285
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:58:56.979892 2347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:58:56.980207 2347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:58:56.980371 2347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:58:56.988507 2347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:58:56.988651 2347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:58:56.988830 2347 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 02:58:56.988911 2347 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:58:56.989030 2347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:58:56.999213 2347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:58:57.000488 2347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:58:57.001931 2347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:58:57.014794 2352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:57.014772 2353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:57.014956 2355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:58:57.016014 2347 server_base.cc:1034] running on GCE node
I20260620 02:58:57.016682 2347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:58:57.017693 2347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:58:57.018918 2347 hybrid_clock.cc:648] HybridClock initialized: now 1781924337018866 us; error 60 us; skew 500 ppm
I20260620 02:58:57.019244 2347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:58:57 dist-test-slave-f1j0 krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:58:57.022361 2347 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:58:57.023732 2347 webserver.cc:460] Webserver started at http://127.0.97.2:33209/ using document root <none> and password file <none>
I20260620 02:58:57.024384 2347 fs_manager.cc:359] Metadata directory not provided
I20260620 02:58:57.024480 2347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:58:57.024751 2347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:58:57.027711 2347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "f4b895774d8345309ddebd223feb4401"
format_stamp: "Formatted at 2026-06-20 02:58:57 on dist-test-slave-f1j0"
server_key: "43f8669e6331250edc8602e399aa162c"
server_key_iv: "22742f2dec9b3fe24c371006f6e4f779"
server_key_version: "encryptionkey@0"
I20260620 02:58:57.028460 2347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f4b895774d8345309ddebd223feb4401"
format_stamp: "Formatted at 2026-06-20 02:58:57 on dist-test-slave-f1j0"
server_key: "43f8669e6331250edc8602e399aa162c"
server_key_iv: "22742f2dec9b3fe24c371006f6e4f779"
server_key_version: "encryptionkey@0"
I20260620 02:58:57.034291 2347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:58:57.038434 2362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:57.039951 2347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260620 02:58:57.040124 2347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "f4b895774d8345309ddebd223feb4401"
format_stamp: "Formatted at 2026-06-20 02:58:57 on dist-test-slave-f1j0"
server_key: "43f8669e6331250edc8602e399aa162c"
server_key_iv: "22742f2dec9b3fe24c371006f6e4f779"
server_key_version: "encryptionkey@0"
I20260620 02:58:57.040330 2347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:58:57.056702 2347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:58:57.060529 2347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:58:57.060773 2347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:58:57.063319 2347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:58:57.063478 2347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:57.063625 2347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:58:57.063726 2347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:58:57 dist-test-slave-f1j0 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:57.088122 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.064064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36615 (local address 127.0.97.62:40285)
0620 02:58:57.064258 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:57.064264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:57.065734 (+ 1470us) server_negotiation.cc:407] Connection header received
0620 02:58:57.067860 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:57.067865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:57.067952 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:57.068084 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:57.070262 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.071399 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.074005 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.074573 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.074703 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.079871 (+ 5168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:57.079880 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:57.079888 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:57.079951 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:57.083982 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:57.085116 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:57.085122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:57.085126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:57.085284 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:57.085876 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:57.085884 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:57.085890 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:57.086296 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:57.086475 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:57.087499 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0620 02:58:57.087744 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:58:57.089870 2371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.064648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40285 (local address 127.0.97.2:36615)
0620 02:58:57.065531 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:57.065608 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:57.067572 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:57.068323 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:57.068331 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:57.069082 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:57.070066 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.070094 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.071541 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.071548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.073853 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.073864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.075025 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.075031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.075274 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.076137 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:57.076177 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:57.079511 (+ 3334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:57.084235 (+ 4724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:57.084249 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:57.084274 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:57.084962 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:57.085395 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:57.085404 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:57.085407 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:57.085725 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:57.086468 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:57.086478 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:57.087008 (+ 530us) client_negotiation.cc:769] Sending connection context
0620 02:58:57.087415 (+ 407us) client_negotiation.cc:240] Negotiation successful
0620 02:58:57.088073 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":399,"threads_started":1}
I20260620 02:58:57.106721 2347 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32897
I20260620 02:58:57.106758 2481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32897 every 8 connection(s)
I20260620 02:58:57.108044 2347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:58:57.110338 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.096825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40285 (local address 127.0.97.2:59175)
0620 02:58:57.097435 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:57.097449 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:57.097634 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:57.098123 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:57.098126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:57.098840 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:57.099333 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.099343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.100625 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.100636 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.101975 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.101986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.102803 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.102810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.102945 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.103568 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:57.103585 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:57.104248 (+ 663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:57.107416 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:57.107422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:57.107427 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:57.107805 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:57.108255 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:57.108261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:57.108265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:57.108412 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:57.109117 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:57.109124 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:57.109353 (+ 229us) client_negotiation.cc:769] Sending connection context
0620 02:58:57.109474 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 02:58:57.109867 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":367,"threads_started":1}
I20260620 02:58:57.110308 2482 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40285
I20260620 02:58:57.110569 2482 heartbeater.cc:464] Registering TS with master...
I20260620 02:58:57.110731 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.096772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59175 (local address 127.0.97.62:40285)
0620 02:58:57.096928 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:57.096932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:57.097540 (+ 608us) server_negotiation.cc:407] Connection header received
0620 02:58:57.097814 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:57.097821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:57.097905 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:57.097972 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:57.099470 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.100496 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.102120 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.102539 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.102673 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.104420 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:57.104426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:57.104431 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:57.104481 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:57.107293 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:57.107919 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:57.107926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:57.107929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:57.108070 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:57.108547 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:57.108555 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:57.108559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:57.108969 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:57.110020 (+ 1051us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:57.110162 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 02:58:57.110375 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 02:58:57.111352 2482 heartbeater.cc:511] Master 127.0.97.62:40285 requested a full tablet report, sending...
I20260620 02:58:57.112833 2137 ts_manager.cc:194] Registered new tserver with Master: f4b895774d8345309ddebd223feb4401 (127.0.97.2:32897)
I20260620 02:58:57.113934 2137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36615
I20260620 02:58:57.117333 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 2347
I20260620 02:58:57.117535 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:58:57.117990 388 external_mini_cluster.cc:1428] Setting key 69d24cb4491b0f24f6ac28c9b3803c06
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:58:57.157383 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40285
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:58:57.471393 2486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:58:57.471675 2486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:58:57.471777 2486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:58:57.479313 2486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:58:57.479483 2486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:58:57.479568 2486 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 02:58:57.479620 2486 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:58:57.479686 2486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:58:57.489455 2486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:58:57.490990 2486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:58:57.492241 2486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:58:57.505313 2491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:57.505417 2492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:57.505646 2494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:58:57.506634 2486 server_base.cc:1034] running on GCE node
I20260620 02:58:57.507246 2486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:58:57.508234 2486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:58:57.509503 2486 hybrid_clock.cc:648] HybridClock initialized: now 1781924337509458 us; error 61 us; skew 500 ppm
I20260620 02:58:57.509938 2486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:58:57 dist-test-slave-f1j0 krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:58:57.513164 2486 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:58:57.514714 2486 webserver.cc:460] Webserver started at http://127.0.97.3:42631/ using document root <none> and password file <none>
I20260620 02:58:57.515383 2486 fs_manager.cc:359] Metadata directory not provided
I20260620 02:58:57.515530 2486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:58:57.515841 2486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:58:57.518733 2486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "a13dae5378be431eaacbde2403e5be15"
format_stamp: "Formatted at 2026-06-20 02:58:57 on dist-test-slave-f1j0"
server_key: "34927c96fe45547677554af683652ddb"
server_key_iv: "b46509f697586fbb23bd7f7540e2e739"
server_key_version: "encryptionkey@0"
I20260620 02:58:57.519502 2486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a13dae5378be431eaacbde2403e5be15"
format_stamp: "Formatted at 2026-06-20 02:58:57 on dist-test-slave-f1j0"
server_key: "34927c96fe45547677554af683652ddb"
server_key_iv: "b46509f697586fbb23bd7f7540e2e739"
server_key_version: "encryptionkey@0"
I20260620 02:58:57.524772 2486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 02:58:57.529471 2501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:57.530893 2486 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:58:57.531031 2486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "a13dae5378be431eaacbde2403e5be15"
format_stamp: "Formatted at 2026-06-20 02:58:57 on dist-test-slave-f1j0"
server_key: "34927c96fe45547677554af683652ddb"
server_key_iv: "b46509f697586fbb23bd7f7540e2e739"
server_key_version: "encryptionkey@0"
I20260620 02:58:57.531217 2486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:58:57.547884 2486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:58:57.552444 2486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:58:57.552681 2486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:58:57.555409 2486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:58:57.555555 2486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:57.555694 2486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:58:57.555794 2486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:58:57 dist-test-slave-f1j0 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:57.579967 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.556131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39325 (local address 127.0.97.62:40285)
0620 02:58:57.556323 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:57.556330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:57.557683 (+ 1353us) server_negotiation.cc:407] Connection header received
0620 02:58:57.559870 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:57.559875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:57.559960 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:57.560078 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:57.562671 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.563772 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.566377 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.567012 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.567173 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.572180 (+ 5007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:57.572185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:57.572188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:57.572219 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:57.575973 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:57.576933 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:57.576939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:57.576942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:57.577114 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:57.577725 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:57.577733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:57.577737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:57.578158 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:57.578357 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:57.579115 (+ 758us) server_negotiation.cc:299] Negotiation successful
0620 02:58:57.579435 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:58:57.580565 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.556641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40285 (local address 127.0.97.3:39325)
0620 02:58:57.557502 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:57.557554 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:57.559588 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:57.560320 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:57.560328 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:57.561240 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:57.562416 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.562448 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.563927 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.563933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.566230 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.566241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.567288 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.567296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.567559 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.568448 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:57.568487 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:57.571916 (+ 3429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:57.576126 (+ 4210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:57.576141 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:57.576159 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:57.576780 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:57.577228 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:57.577234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:57.577237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:57.577562 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:57.578306 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:57.578312 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:57.578726 (+ 414us) client_negotiation.cc:769] Sending connection context
0620 02:58:57.578955 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 02:58:57.579332 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":413,"threads_started":1}
I20260620 02:58:57.602222 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.586180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41353 (local address 127.0.97.62:40285)
0620 02:58:57.586363 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:57.586368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:57.587154 (+ 786us) server_negotiation.cc:407] Connection header received
0620 02:58:57.587333 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:57.587337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:57.587403 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:57.587492 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:57.588917 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.589707 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.592309 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.592995 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.593162 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.595575 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:57.595581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:57.595586 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:57.595631 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:57.599134 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:57.600134 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:57.600138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:57.600141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:57.600288 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:57.600703 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:57.600706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:57.600709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:57.600942 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:57.601105 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:57.601701 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 02:58:57.601898 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:58:57.602677 2573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.586249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40285 (local address 127.0.97.3:41353)
0620 02:58:57.587019 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:57.587044 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:57.587216 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:57.587567 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:57.587574 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:57.588310 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:57.588751 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.588767 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.589864 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.589871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.592151 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.592170 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.593222 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.593232 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.593480 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.594267 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:57.594295 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:57.595395 (+ 1100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:57.599331 (+ 3936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:57.599336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:57.599345 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:57.600023 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:57.600416 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:57.600421 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:57.600424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:57.600591 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:57.601090 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:57.601098 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:57.601360 (+ 262us) client_negotiation.cc:769] Sending connection context
0620 02:58:57.601595 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 02:58:57.602025 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":408,"threads_started":1}
I20260620 02:58:57.613946 2486 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34871
I20260620 02:58:57.614027 2620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34871 every 8 connection(s)
I20260620 02:58:57.615175 2486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:58:57.617256 2621 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40285
I20260620 02:58:57.617446 2621 heartbeater.cc:464] Registering TS with master...
I20260620 02:58:57.618089 2621 heartbeater.cc:511] Master 127.0.97.62:40285 requested a full tablet report, sending...
I20260620 02:58:57.619338 2137 ts_manager.cc:194] Registered new tserver with Master: a13dae5378be431eaacbde2403e5be15 (127.0.97.3:34871)
I20260620 02:58:57.620147 2137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39325
I20260620 02:58:57.624276 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 2486
I20260620 02:58:57.624486 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:58:57.624903 388 external_mini_cluster.cc:1428] Setting key 1eb856bcd46f7e5c5d7f60dca94f07f1
I20260620 02:58:57.627668 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:58:57.633026 2343 heartbeater.cc:503] Master 127.0.97.62:40285 was elected leader, sending a full tablet report...
Jun 20 02:58:57 dist-test-slave-f1j0 krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:58:57 dist-test-slave-f1j0 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:57.672089 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.659433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.0.97.62:40285)
0620 02:58:57.659660 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:57.659666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:57.659802 (+ 136us) server_negotiation.cc:407] Connection header received
0620 02:58:57.659945 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:57.659949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:57.660025 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:57.660110 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:57.661379 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.662123 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.663288 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.663671 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.663757 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.667318 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:57.667333 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:57.667336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:57.667373 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:57.669926 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:57.670473 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:57.670477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:57.670479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:57.670580 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:57.670947 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:57.670950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:57.670952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:57.671176 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:57.671349 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:57.671631 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 02:58:57.671778 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:58:57.676024 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:58:57.682898 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:58:57.716578 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.695193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.0.97.1:38291)
0620 02:58:57.709219 (+ 14026us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:57.709231 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:57.709269 (+ 38us) server_negotiation.cc:407] Connection header received
0620 02:58:57.709399 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:57.709412 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:57.709777 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:57.710551 (+ 774us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:57.710691 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.712165 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.714163 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.715473 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.715606 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.715810 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:57.716111 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 02:58:57.716261 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13828,"thread_start_us":13670,"threads_started":1}
I20260620 02:58:57.716614 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.694897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38291 (local address 127.0.0.1:39046)
0620 02:58:57.707087 (+ 12190us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:57.707110 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:57.707244 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:57.710032 (+ 2788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:57.710042 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:57.710088 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:57.710397 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.710406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.712318 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.712322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.713970 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.713985 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.715655 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.715661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.715863 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.715921 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 02:58:57.716047 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 02:58:57.716239 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12041,"thread_start_us":19099,"threads_started":1}
I20260620 02:58:57.719323 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.695261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43476 (local address 127.0.97.3:34871)
0620 02:58:57.710269 (+ 15008us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:57.710282 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:57.710323 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:58:57.710441 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:57.710452 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:57.710759 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:57.711008 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:57.711989 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.714009 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.716357 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.718244 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.718385 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.718618 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:57.718861 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 02:58:57.719003 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14784,"thread_start_us":14561,"threads_started":1}
I20260620 02:58:57.719362 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.694864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34871 (local address 127.0.0.1:43476)
0620 02:58:57.705271 (+ 10407us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:57.705332 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:57.705546 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:57.710995 (+ 5449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:57.710999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:57.711023 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:57.711393 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.711403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.714159 (+ 2756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.714165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.716189 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.716201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.718449 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.718455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.718684 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.718708 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 02:58:57.718815 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 02:58:57.718937 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10057,"mutex_wait_us":24,"thread_start_us":12474,"threads_started":1}
I20260620 02:58:57.721186 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.697426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32897 (local address 127.0.0.1:49138)
0620 02:58:57.713775 (+ 16349us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:57.713796 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:57.713915 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:57.714502 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:57.714506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:57.714530 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:57.714955 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.714966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.716635 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.716642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.719037 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.719050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.720656 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.720660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.720774 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.720795 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 02:58:57.720868 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 02:58:57.720946 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16217,"mutex_wait_us":25,"thread_start_us":7304,"threads_started":1}
I20260620 02:58:57.720671 2273 tablet_service.cc:1467] Processing CreateTablet for tablet 98503fc8966b48d097387ff66b118af4 (DEFAULT_TABLE table=test-table [id=1a2e1e83a4bb46ef89f0359f12906803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:58:57.721364 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.697978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49138 (local address 127.0.97.2:32897)
0620 02:58:57.706221 (+ 8243us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:57.706235 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:57.713921 (+ 7686us) server_negotiation.cc:407] Connection header received
0620 02:58:57.714037 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:57.714045 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:57.714301 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:57.714487 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:57.715144 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.716463 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.719221 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.720462 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.720589 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.720777 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:57.720932 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 02:58:57.721084 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8035,"thread_start_us":7884,"threads_started":1}
I20260620 02:58:57.722247 2273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98503fc8966b48d097387ff66b118af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:57.724274 2412 tablet_service.cc:1467] Processing CreateTablet for tablet 98503fc8966b48d097387ff66b118af4 (DEFAULT_TABLE table=test-table [id=1a2e1e83a4bb46ef89f0359f12906803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:58:57.724895 2551 tablet_service.cc:1467] Processing CreateTablet for tablet 98503fc8966b48d097387ff66b118af4 (DEFAULT_TABLE table=test-table [id=1a2e1e83a4bb46ef89f0359f12906803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:58:57.725716 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98503fc8966b48d097387ff66b118af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:57.726231 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98503fc8966b48d097387ff66b118af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:57.736519 2640 tablet_bootstrap.cc:492] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401: Bootstrap starting.
I20260620 02:58:57.739506 2640 tablet_bootstrap.cc:654] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:57.740320 2641 tablet_bootstrap.cc:492] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4: Bootstrap starting.
I20260620 02:58:57.740839 2640 log.cc:824] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401: Log is configured to *not* fsync() on all Append() calls
I20260620 02:58:57.742244 2642 tablet_bootstrap.cc:492] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15: Bootstrap starting.
I20260620 02:58:57.744027 2640 tablet_bootstrap.cc:492] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401: No bootstrap required, opened a new log
I20260620 02:58:57.744305 2640 ts_tablet_manager.cc:1397] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 02:58:57.744431 2641 tablet_bootstrap.cc:654] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:57.745743 2642 tablet_bootstrap.cc:654] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:57.745757 2641 log.cc:824] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4: Log is configured to *not* fsync() on all Append() calls
I20260620 02:58:57.746832 2642 log.cc:824] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15: Log is configured to *not* fsync() on all Append() calls
I20260620 02:58:57.748653 2641 tablet_bootstrap.cc:492] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4: No bootstrap required, opened a new log
I20260620 02:58:57.748927 2641 ts_tablet_manager.cc:1397] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 02:58:57.749719 2642 tablet_bootstrap.cc:492] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15: No bootstrap required, opened a new log
I20260620 02:58:57.749956 2642 ts_tablet_manager.cc:1397] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260620 02:58:57.752261 2640 raft_consensus.cc:348] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } } peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } }
I20260620 02:58:57.752606 2640 raft_consensus.cc:374] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:57.752712 2640 raft_consensus.cc:729] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b895774d8345309ddebd223feb4401, State: Initialized, Role: FOLLOWER
I20260620 02:58:57.753319 2640 consensus_queue.cc:260] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } } peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } }
I20260620 02:58:57.754338 2482 heartbeater.cc:503] Master 127.0.97.62:40285 was elected leader, sending a full tablet report...
I20260620 02:58:57.754968 2640 ts_tablet_manager.cc:1428] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401: Time spent starting tablet: real 0.011s user 0.003s sys 0.009s
I20260620 02:58:57.759378 2641 raft_consensus.cc:348] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } } peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } }
I20260620 02:58:57.759804 2641 raft_consensus.cc:374] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:57.759912 2641 raft_consensus.cc:729] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 561036292d05476c840ea0e9636a1bf4, State: Initialized, Role: FOLLOWER
I20260620 02:58:57.760505 2641 consensus_queue.cc:260] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } } peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } }
I20260620 02:58:57.760692 2642 raft_consensus.cc:348] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } } peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } }
I20260620 02:58:57.761122 2642 raft_consensus.cc:374] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:57.761221 2642 raft_consensus.cc:729] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a13dae5378be431eaacbde2403e5be15, State: Initialized, Role: FOLLOWER
I20260620 02:58:57.761783 2642 consensus_queue.cc:260] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } } peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } }
I20260620 02:58:57.762261 2641 ts_tablet_manager.cc:1428] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
I20260620 02:58:57.762938 2621 heartbeater.cc:503] Master 127.0.97.62:40285 was elected leader, sending a full tablet report...
I20260620 02:58:57.763490 2642 ts_tablet_manager.cc:1428] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
I20260620 02:58:57.799712 2646 raft_consensus.cc:482] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:58:57.800016 2646 raft_consensus.cc:504] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } } peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } }
I20260620 02:58:57.801705 2646 leader_election.cc:288] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a13dae5378be431eaacbde2403e5be15 (127.0.97.3:34871), 561036292d05476c840ea0e9636a1bf4 (127.0.97.1:38291)
I20260620 02:58:57.808264 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.802084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50841 (local address 127.0.97.3:34871)
0620 02:58:57.802295 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:57.802302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:57.802957 (+ 655us) server_negotiation.cc:407] Connection header received
0620 02:58:57.803188 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:57.803195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:57.803257 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:57.803353 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:57.803893 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.804649 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.806545 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.807303 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.807465 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.807579 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:57.807869 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 02:58:57.808001 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:58:57.808310 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.802319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56655 (local address 127.0.97.1:38291)
0620 02:58:57.802533 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:57.802538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:57.803136 (+ 598us) server_negotiation.cc:407] Connection header received
0620 02:58:57.803331 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:57.803335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:57.803400 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:57.803492 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:57.804118 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.804823 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.806655 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.807406 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.807549 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.807650 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:57.807923 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 02:58:57.808055 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:58:57.808601 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.801841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34871 (local address 127.0.97.2:50841)
0620 02:58:57.802858 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:57.802881 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:57.803028 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:57.803366 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:57.803371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:57.803392 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:57.803746 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.803754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.804774 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.804782 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.806386 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.806397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.807545 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.807550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.807717 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.807739 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 02:58:57.807847 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 02:58:57.807972 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":2,"thread_start_us":642,"threads_started":1}
I20260620 02:58:57.808621 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.802247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38291 (local address 127.0.97.2:56655)
0620 02:58:57.803031 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:57.803044 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:57.803183 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:57.803509 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:57.803513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:57.803598 (+ 85us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:57.803951 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.803959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.804958 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.804963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.806502 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.806512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.807624 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.807635 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.807776 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.807798 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 02:58:57.807898 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 02:58:57.808031 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":45,"spinlock_wait_cycles":30464,"thread_start_us":465,"threads_started":1}
I20260620 02:58:57.809667 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98503fc8966b48d097387ff66b118af4" candidate_uuid: "f4b895774d8345309ddebd223feb4401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "561036292d05476c840ea0e9636a1bf4" is_pre_election: true
I20260620 02:58:57.809667 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98503fc8966b48d097387ff66b118af4" candidate_uuid: "f4b895774d8345309ddebd223feb4401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a13dae5378be431eaacbde2403e5be15" is_pre_election: true
I20260620 02:58:57.810217 2576 raft_consensus.cc:2454] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4b895774d8345309ddebd223feb4401 in term 0.
I20260620 02:58:57.810231 2298 raft_consensus.cc:2454] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4b895774d8345309ddebd223feb4401 in term 0.
I20260620 02:58:57.810933 2366 leader_election.cc:302] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 561036292d05476c840ea0e9636a1bf4, a13dae5378be431eaacbde2403e5be15, f4b895774d8345309ddebd223feb4401; no voters:
I20260620 02:58:57.811353 2646 raft_consensus.cc:2781] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:58:57.811491 2646 raft_consensus.cc:482] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:58:57.811596 2646 raft_consensus.cc:3037] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:57.813735 2646 raft_consensus.cc:504] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } } peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } }
I20260620 02:58:57.814250 2646 leader_election.cc:288] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [CANDIDATE]: Term 1 election: Requested vote from peers a13dae5378be431eaacbde2403e5be15 (127.0.97.3:34871), 561036292d05476c840ea0e9636a1bf4 (127.0.97.1:38291)
I20260620 02:58:57.814636 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98503fc8966b48d097387ff66b118af4" candidate_uuid: "f4b895774d8345309ddebd223feb4401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a13dae5378be431eaacbde2403e5be15"
I20260620 02:58:57.814720 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98503fc8966b48d097387ff66b118af4" candidate_uuid: "f4b895774d8345309ddebd223feb4401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "561036292d05476c840ea0e9636a1bf4"
I20260620 02:58:57.814816 2576 raft_consensus.cc:3037] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:57.814956 2298 raft_consensus.cc:3037] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:57.817327 2576 raft_consensus.cc:2454] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4b895774d8345309ddebd223feb4401 in term 1.
I20260620 02:58:57.817502 2298 raft_consensus.cc:2454] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4b895774d8345309ddebd223feb4401 in term 1.
I20260620 02:58:57.817730 2365 leader_election.cc:302] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a13dae5378be431eaacbde2403e5be15, f4b895774d8345309ddebd223feb4401; no voters:
I20260620 02:58:57.818035 2646 raft_consensus.cc:2781] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:58:57.818425 2646 raft_consensus.cc:686] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [term 1 LEADER]: Becoming Leader. State: Replica: f4b895774d8345309ddebd223feb4401, State: Running, Role: LEADER
I20260620 02:58:57.818943 2646 consensus_queue.cc:237] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } } peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } }
I20260620 02:58:57.823190 2136 catalog_manager.cc:5495] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4b895774d8345309ddebd223feb4401 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f4b895774d8345309ddebd223feb4401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } health_report { overall_health: HEALTHY } } }
W20260620 02:58:57.860608 2483 tablet.cc:2369] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:58:57.866758 2622 tablet.cc:2369] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:58:57.875878 2344 tablet.cc:2369] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 02:58:57 dist-test-slave-f1j0 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:57.885136 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.871938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40285 (local address 127.0.0.1:41258)
0620 02:58:57.872566 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:57.872582 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:57.872718 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:57.873003 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:57.873006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:57.873735 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:57.873987 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.873996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.875034 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.875037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.876364 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:57.876376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.877033 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.877037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:57.877163 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.877849 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:57.877872 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:57.880207 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:57.882927 (+ 2720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:57.882936 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:57.882942 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:57.883430 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:57.883767 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:57.883770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:57.883772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:57.883890 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:57.884333 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:57.884339 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:57.884476 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 02:58:57.884610 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 02:58:57.884798 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":331,"threads_started":1}
I20260620 02:58:57.885196 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:57.871914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41258 (local address 127.0.97.62:40285)
0620 02:58:57.872120 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:57.872128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:57.872683 (+ 555us) server_negotiation.cc:407] Connection header received
0620 02:58:57.872818 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:57.872822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:57.872880 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:57.872961 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:57.874138 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.874925 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.876508 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:57.876886 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:57.876974 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:57.880385 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:57.880390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:57.880393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:57.880419 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:57.882819 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:57.883527 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:57.883531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:57.883533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:57.883675 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:57.883981 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:57.883984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:57.883986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:57.884232 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:57.884331 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:57.884683 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 02:58:57.884903 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":67}
I20260620 02:58:57.890215 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41258:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 02:58:57.891381 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:58:57.898195 2412 tablet_service.cc:1467] Processing CreateTablet for tablet 3d3e3fec60204fd383361d7b9cbe06d0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0a808fe10d5c4267b28b67e4b567519a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:58:57.897985 2273 tablet_service.cc:1467] Processing CreateTablet for tablet 3d3e3fec60204fd383361d7b9cbe06d0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0a808fe10d5c4267b28b67e4b567519a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:58:57.898196 2551 tablet_service.cc:1467] Processing CreateTablet for tablet 3d3e3fec60204fd383361d7b9cbe06d0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0a808fe10d5c4267b28b67e4b567519a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:58:57.898963 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d3e3fec60204fd383361d7b9cbe06d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:57.898967 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d3e3fec60204fd383361d7b9cbe06d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:57.898967 2273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d3e3fec60204fd383361d7b9cbe06d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:57.904505 2640 tablet_bootstrap.cc:492] T 3d3e3fec60204fd383361d7b9cbe06d0 P f4b895774d8345309ddebd223feb4401: Bootstrap starting.
I20260620 02:58:57.905885 2642 tablet_bootstrap.cc:492] T 3d3e3fec60204fd383361d7b9cbe06d0 P a13dae5378be431eaacbde2403e5be15: Bootstrap starting.
I20260620 02:58:57.906251 2640 tablet_bootstrap.cc:654] T 3d3e3fec60204fd383361d7b9cbe06d0 P f4b895774d8345309ddebd223feb4401: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:57.906363 2641 tablet_bootstrap.cc:492] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4: Bootstrap starting.
I20260620 02:58:57.907933 2642 tablet_bootstrap.cc:654] T 3d3e3fec60204fd383361d7b9cbe06d0 P a13dae5378be431eaacbde2403e5be15: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:57.908031 2641 tablet_bootstrap.cc:654] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:57.910102 2642 tablet_bootstrap.cc:492] T 3d3e3fec60204fd383361d7b9cbe06d0 P a13dae5378be431eaacbde2403e5be15: No bootstrap required, opened a new log
I20260620 02:58:57.910257 2642 ts_tablet_manager.cc:1397] T 3d3e3fec60204fd383361d7b9cbe06d0 P a13dae5378be431eaacbde2403e5be15: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260620 02:58:57.910920 2640 tablet_bootstrap.cc:492] T 3d3e3fec60204fd383361d7b9cbe06d0 P f4b895774d8345309ddebd223feb4401: No bootstrap required, opened a new log
I20260620 02:58:57.911077 2640 ts_tablet_manager.cc:1397] T 3d3e3fec60204fd383361d7b9cbe06d0 P f4b895774d8345309ddebd223feb4401: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260620 02:58:57.910991 2642 raft_consensus.cc:348] T 3d3e3fec60204fd383361d7b9cbe06d0 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:58:57.911268 2642 raft_consensus.cc:374] T 3d3e3fec60204fd383361d7b9cbe06d0 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:57.911365 2642 raft_consensus.cc:729] T 3d3e3fec60204fd383361d7b9cbe06d0 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a13dae5378be431eaacbde2403e5be15, State: Initialized, Role: FOLLOWER
I20260620 02:58:57.911589 2642 consensus_queue.cc:260] T 3d3e3fec60204fd383361d7b9cbe06d0 P a13dae5378be431eaacbde2403e5be15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:58:57.911804 2640 raft_consensus.cc:348] T 3d3e3fec60204fd383361d7b9cbe06d0 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:58:57.912024 2640 raft_consensus.cc:374] T 3d3e3fec60204fd383361d7b9cbe06d0 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:57.912111 2640 raft_consensus.cc:729] T 3d3e3fec60204fd383361d7b9cbe06d0 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b895774d8345309ddebd223feb4401, State: Initialized, Role: FOLLOWER
I20260620 02:58:57.912153 2642 ts_tablet_manager.cc:1428] T 3d3e3fec60204fd383361d7b9cbe06d0 P a13dae5378be431eaacbde2403e5be15: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 02:58:57.912019 2648 tablet_replica.cc:439] T 3d3e3fec60204fd383361d7b9cbe06d0 P a13dae5378be431eaacbde2403e5be15: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } } }
I20260620 02:58:57.912393 2648 tablet_replica.cc:442] T 3d3e3fec60204fd383361d7b9cbe06d0 P a13dae5378be431eaacbde2403e5be15: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:58:57.912309 2640 consensus_queue.cc:260] T 3d3e3fec60204fd383361d7b9cbe06d0 P f4b895774d8345309ddebd223feb4401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:58:57.912748 2646 tablet_replica.cc:439] T 3d3e3fec60204fd383361d7b9cbe06d0 P f4b895774d8345309ddebd223feb4401: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } } }
I20260620 02:58:57.912935 2640 ts_tablet_manager.cc:1428] T 3d3e3fec60204fd383361d7b9cbe06d0 P f4b895774d8345309ddebd223feb4401: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 02:58:57.913035 2646 tablet_replica.cc:442] T 3d3e3fec60204fd383361d7b9cbe06d0 P f4b895774d8345309ddebd223feb4401: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:58:57.913625 2641 tablet_bootstrap.cc:492] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4: No bootstrap required, opened a new log
I20260620 02:58:57.913764 2641 ts_tablet_manager.cc:1397] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20260620 02:58:57.914484 2641 raft_consensus.cc:348] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:58:57.914713 2641 raft_consensus.cc:374] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:57.914810 2641 raft_consensus.cc:729] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 561036292d05476c840ea0e9636a1bf4, State: Initialized, Role: FOLLOWER
I20260620 02:58:57.915020 2641 consensus_queue.cc:260] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:58:57.915728 2647 tablet_replica.cc:439] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } } }
I20260620 02:58:57.916090 2647 tablet_replica.cc:442] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:58:57.916643 2641 ts_tablet_manager.cc:1428] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 02:58:58.099113 2647 raft_consensus.cc:482] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:58:58.099336 2647 raft_consensus.cc:504] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:58:58.100781 2647 leader_election.cc:288] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4b895774d8345309ddebd223feb4401 (127.0.97.2:32897), a13dae5378be431eaacbde2403e5be15 (127.0.97.3:34871)
I20260620 02:58:58.108268 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:58.101154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58179 (local address 127.0.97.3:34871)
0620 02:58:58.101438 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:58.101446 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:58.101833 (+ 387us) server_negotiation.cc:407] Connection header received
0620 02:58:58.102135 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:58.102139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:58.102232 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:58.102332 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:58.103394 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.104159 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.106331 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.107172 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.107312 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:58.107410 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:58.107802 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 02:58:58.107943 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260620 02:58:58.109078 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:58.101741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47033 (local address 127.0.97.2:32897)
0620 02:58:58.101950 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:58.101957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:58.102202 (+ 245us) server_negotiation.cc:407] Connection header received
0620 02:58:58.102301 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:58.102309 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:58.102381 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:58.102469 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:58:58.103368 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.104506 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.106999 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.108117 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.108280 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:58.108380 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:58.108683 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 02:58:58.108805 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:58:58.109084 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:58.101028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34871 (local address 127.0.97.1:58179)
0620 02:58:58.101719 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:58.101743 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:58.101965 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:58.102355 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:58.102358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:58.102376 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:58.103175 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:58.103196 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.104307 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.104312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:58.106180 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:58.106197 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.107390 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.107398 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:58.107605 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:58.107639 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 02:58:58.108446 (+ 807us) client_negotiation.cc:240] Negotiation successful
0620 02:58:58.108656 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":57,"spinlock_wait_cycles":39552,"thread_start_us":424,"threads_started":1}
I20260620 02:58:58.109360 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:58.101140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32897 (local address 127.0.97.1:47033)
0620 02:58:58.102110 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:58.102126 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:58.102210 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:58.102499 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:58.102506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:58.102542 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:58:58.103175 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:58.103190 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.104703 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.104708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:58.106815 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:58.106828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.108329 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.108335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:58.108540 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:58.108564 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 02:58:58.108660 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 02:58:58.108793 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":1,"thread_start_us":644,"threads_started":1}
I20260620 02:58:58.109776 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3e3fec60204fd383361d7b9cbe06d0" candidate_uuid: "561036292d05476c840ea0e9636a1bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a13dae5378be431eaacbde2403e5be15" is_pre_election: true
I20260620 02:58:58.110061 2576 raft_consensus.cc:2454] T 3d3e3fec60204fd383361d7b9cbe06d0 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 561036292d05476c840ea0e9636a1bf4 in term 0.
I20260620 02:58:58.110213 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3e3fec60204fd383361d7b9cbe06d0" candidate_uuid: "561036292d05476c840ea0e9636a1bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b895774d8345309ddebd223feb4401" is_pre_election: true
I20260620 02:58:58.110528 2226 leader_election.cc:302] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 561036292d05476c840ea0e9636a1bf4, a13dae5378be431eaacbde2403e5be15; no voters:
I20260620 02:58:58.110639 2432 raft_consensus.cc:2454] T 3d3e3fec60204fd383361d7b9cbe06d0 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 561036292d05476c840ea0e9636a1bf4 in term 0.
I20260620 02:58:58.110952 2647 raft_consensus.cc:2781] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:58:58.111104 2647 raft_consensus.cc:482] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:58:58.111207 2647 raft_consensus.cc:3037] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:58.114167 2647 raft_consensus.cc:504] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:58:58.115049 2647 leader_election.cc:288] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 [CANDIDATE]: Term 1 election: Requested vote from peers f4b895774d8345309ddebd223feb4401 (127.0.97.2:32897), a13dae5378be431eaacbde2403e5be15 (127.0.97.3:34871)
I20260620 02:58:58.115510 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3e3fec60204fd383361d7b9cbe06d0" candidate_uuid: "561036292d05476c840ea0e9636a1bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a13dae5378be431eaacbde2403e5be15"
I20260620 02:58:58.115747 2576 raft_consensus.cc:3037] T 3d3e3fec60204fd383361d7b9cbe06d0 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:58.118479 2576 raft_consensus.cc:2454] T 3d3e3fec60204fd383361d7b9cbe06d0 P a13dae5378be431eaacbde2403e5be15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 561036292d05476c840ea0e9636a1bf4 in term 1.
I20260620 02:58:58.118850 2226 leader_election.cc:302] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 561036292d05476c840ea0e9636a1bf4, a13dae5378be431eaacbde2403e5be15; no voters:
I20260620 02:58:58.119457 2647 raft_consensus.cc:2781] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:58:58.119503 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3e3fec60204fd383361d7b9cbe06d0" candidate_uuid: "561036292d05476c840ea0e9636a1bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b895774d8345309ddebd223feb4401"
I20260620 02:58:58.119714 2432 raft_consensus.cc:3037] T 3d3e3fec60204fd383361d7b9cbe06d0 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:58.119939 2647 raft_consensus.cc:686] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 561036292d05476c840ea0e9636a1bf4, State: Running, Role: LEADER
I20260620 02:58:58.120764 2647 consensus_queue.cc:237] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:58:58.122886 2432 raft_consensus.cc:2454] T 3d3e3fec60204fd383361d7b9cbe06d0 P f4b895774d8345309ddebd223feb4401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 561036292d05476c840ea0e9636a1bf4 in term 1.
I20260620 02:58:58.125036 2664 tablet_replica.cc:439] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4: TxnStatusTablet state changed. Reason: New leader 561036292d05476c840ea0e9636a1bf4. Latest consensus state: current_term: 1 leader_uuid: "561036292d05476c840ea0e9636a1bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } } }
I20260620 02:58:58.125342 2664 tablet_replica.cc:442] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:58:58.126052 2666 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 02:58:58.126344 2666 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 02:58:58.127025 2135 catalog_manager.cc:5495] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 561036292d05476c840ea0e9636a1bf4 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "561036292d05476c840ea0e9636a1bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } health_report { overall_health: UNKNOWN } } }
I20260620 02:58:58.152563 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:58.145368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38291 (local address 127.0.0.1:39060)
0620 02:58:58.145772 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:58.145792 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:58.145957 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:58.146397 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:58.146402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:58.146415 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:58:58.146614 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:58.146626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.147654 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.147660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:58.149313 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:58.149328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.150226 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.150233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:58.150445 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:58.150524 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:58:58.151880 (+ 1356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:58:58.151905 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 02:58:58.152024 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 02:58:58.152206 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234}
I20260620 02:58:58.152597 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:58.145362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.0.97.1:38291)
0620 02:58:58.145521 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:58.145525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:58.145882 (+ 357us) server_negotiation.cc:407] Connection header received
0620 02:58:58.146145 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:58.146149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:58.146234 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:58.146344 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:58:58.146739 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.147520 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.149476 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.150055 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.150166 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:58.150930 (+ 764us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:58:58.151719 (+ 789us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:58:58.151929 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:58.152081 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 02:58:58.152205 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 02:58:58.160995 2576 raft_consensus.cc:1261] T 3d3e3fec60204fd383361d7b9cbe06d0 P a13dae5378be431eaacbde2403e5be15 [term 1 FOLLOWER]: Refusing update from remote peer 561036292d05476c840ea0e9636a1bf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:58:58.161015 2432 raft_consensus.cc:1261] T 3d3e3fec60204fd383361d7b9cbe06d0 P f4b895774d8345309ddebd223feb4401 [term 1 FOLLOWER]: Refusing update from remote peer 561036292d05476c840ea0e9636a1bf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:58:58.162031 2664 consensus_queue.cc:1035] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:58:58.162379 2647 consensus_queue.cc:1035] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:58:58.171208 2648 tablet_replica.cc:439] T 3d3e3fec60204fd383361d7b9cbe06d0 P a13dae5378be431eaacbde2403e5be15: TxnStatusTablet state changed. Reason: New leader 561036292d05476c840ea0e9636a1bf4. Latest consensus state: current_term: 1 leader_uuid: "561036292d05476c840ea0e9636a1bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } } }
I20260620 02:58:58.171500 2648 tablet_replica.cc:442] T 3d3e3fec60204fd383361d7b9cbe06d0 P a13dae5378be431eaacbde2403e5be15: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:58:58.177284 2669 mvcc.cc:205] Tried to move back new op lower bound from 7298762089097723904 to 7298762088947843072. Current Snapshot: MvccSnapshot[applied={T|T < 7298762089097723904}]
I20260620 02:58:58.178665 2647 tablet_replica.cc:439] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "561036292d05476c840ea0e9636a1bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } } }
I20260620 02:58:58.179061 2647 tablet_replica.cc:442] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:58:58.183234 2648 tablet_replica.cc:439] T 3d3e3fec60204fd383361d7b9cbe06d0 P a13dae5378be431eaacbde2403e5be15: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "561036292d05476c840ea0e9636a1bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } } }
I20260620 02:58:58.183466 2648 tablet_replica.cc:442] T 3d3e3fec60204fd383361d7b9cbe06d0 P a13dae5378be431eaacbde2403e5be15: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:58:58.184587 2646 tablet_replica.cc:439] T 3d3e3fec60204fd383361d7b9cbe06d0 P f4b895774d8345309ddebd223feb4401: TxnStatusTablet state changed. Reason: New leader 561036292d05476c840ea0e9636a1bf4. Latest consensus state: current_term: 1 leader_uuid: "561036292d05476c840ea0e9636a1bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } } }
I20260620 02:58:58.184890 2646 tablet_replica.cc:442] T 3d3e3fec60204fd383361d7b9cbe06d0 P f4b895774d8345309ddebd223feb4401: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:58:58.187206 2664 tablet_replica.cc:439] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "561036292d05476c840ea0e9636a1bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } } }
I20260620 02:58:58.187466 2664 tablet_replica.cc:442] T 3d3e3fec60204fd383361d7b9cbe06d0 P 561036292d05476c840ea0e9636a1bf4: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:58:58.193712 2671 mvcc.cc:205] Tried to move back new op lower bound from 7298762089097723904 to 7298762088947843072. Current Snapshot: MvccSnapshot[applied={T|T < 7298762089097723904}]
I20260620 02:58:58.202313 2646 tablet_replica.cc:439] T 3d3e3fec60204fd383361d7b9cbe06d0 P f4b895774d8345309ddebd223feb4401: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "561036292d05476c840ea0e9636a1bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } } }
I20260620 02:58:58.202621 2646 tablet_replica.cc:442] T 3d3e3fec60204fd383361d7b9cbe06d0 P f4b895774d8345309ddebd223feb4401: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:58:58.209038 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:58.193221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49152 (local address 127.0.97.2:32897)
0620 02:58:58.193413 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:58.193422 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:58.202434 (+ 9012us) server_negotiation.cc:407] Connection header received
0620 02:58:58.202789 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:58.202795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:58.202904 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:58.203047 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:58:58.203799 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.204912 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.207019 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.207408 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.207539 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:58.207930 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:58:58.208461 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:58:58.208609 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:58.208685 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 02:58:58.208761 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 02:58:58.227938 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:58.220616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47435 (local address 127.0.97.1:38291)
0620 02:58:58.220885 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:58.220894 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:58.221772 (+ 878us) server_negotiation.cc:407] Connection header received
0620 02:58:58.221933 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:58.221939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:58.222063 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:58.222172 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:58:58.222755 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.223665 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.225796 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.226586 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.226716 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:58.227193 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:58:58.227317 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:58:58.227415 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:58.227543 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 02:58:58.227615 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260620 02:58:58.228034 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:58.220949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38291 (local address 127.0.97.2:47435)
0620 02:58:58.221648 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:58.221671 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:58.221813 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:58.222195 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:58.222200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:58.222212 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:58:58.222587 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:58.222606 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.223892 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.223898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:58.225617 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:58.225633 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.226743 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.226750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:58.226916 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:58.226982 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:58:58.227431 (+ 449us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:58:58.227443 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 02:58:58.227517 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 02:58:58.227648 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":336,"threads_started":1}
I20260620 02:58:58.238873 2576 raft_consensus.cc:1261] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15 [term 1 FOLLOWER]: Refusing update from remote peer f4b895774d8345309ddebd223feb4401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:58:58.239086 2298 raft_consensus.cc:1261] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4 [term 1 FOLLOWER]: Refusing update from remote peer f4b895774d8345309ddebd223feb4401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:58:58.239491 2651 consensus_queue.cc:1035] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:58:58.239920 2646 consensus_queue.cc:1035] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:58:58.250540 2670 mvcc.cc:205] Tried to move back new op lower bound from 7298762089420218368 to 7298762087710248960. Current Snapshot: MvccSnapshot[applied={T|T < 7298762089420218368}]
I20260620 02:58:58.285254 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:58.278924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54021 (local address 127.0.97.2:32897)
0620 02:58:58.279127 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:58.279133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:58.279322 (+ 189us) server_negotiation.cc:407] Connection header received
0620 02:58:58.279548 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:58.279553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:58.279627 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:58.279716 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:58:58.280331 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.281307 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.283148 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.283809 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.283947 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:58.284419 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:58:58.284542 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:58:58.284654 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:58.284892 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 02:58:58.284967 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 02:58:58.285492 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:58.278515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32897 (local address 127.0.97.1:54021)
0620 02:58:58.279228 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:58.279246 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:58.279403 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:58.279731 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:58.279736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:58.279748 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:58:58.280110 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:58.280121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.281478 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.281489 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:58.282953 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:58.282965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.283988 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.283994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:58.284114 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:58.284220 (+ 106us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:58:58.284767 (+ 547us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:58:58.284776 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 02:58:58.284868 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 02:58:58.285000 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":453,"threads_started":1}
I20260620 02:58:58.326805 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:58:58.327102 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:58:58.330982 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045: Sending DeleteTablet for 3 replicas of tablet 98503fc8966b48d097387ff66b118af4
I20260620 02:58:58.332266 2551 tablet_service.cc:1514] Processing DeleteTablet for tablet 98503fc8966b48d097387ff66b118af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:58:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43476
I20260620 02:58:58.332978 2273 tablet_service.cc:1514] Processing DeleteTablet for tablet 98503fc8966b48d097387ff66b118af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:58:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39046
I20260620 02:58:58.334115 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet 98503fc8966b48d097387ff66b118af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:58:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49138
I20260620 02:58:58.335561 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:58:58.336205 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:58:58.345924 2550 tablet_service.cc:1467] Processing CreateTablet for tablet b091fd521c4d416d87cedd9851f953d5 (DEFAULT_TABLE table=test-table [id=3a1e58582aae456ab9ad0667e2b99504]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:58:58.346030 2272 tablet_service.cc:1467] Processing CreateTablet for tablet b091fd521c4d416d87cedd9851f953d5 (DEFAULT_TABLE table=test-table [id=3a1e58582aae456ab9ad0667e2b99504]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:58:58.346498 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b091fd521c4d416d87cedd9851f953d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:58.346557 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b091fd521c4d416d87cedd9851f953d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:58.350373 2411 tablet_service.cc:1467] Processing CreateTablet for tablet b091fd521c4d416d87cedd9851f953d5 (DEFAULT_TABLE table=test-table [id=3a1e58582aae456ab9ad0667e2b99504]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:58:58.350945 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b091fd521c4d416d87cedd9851f953d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:58.354893 2642 tablet_bootstrap.cc:492] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15: Bootstrap starting.
I20260620 02:58:58.356133 2693 tablet_replica.cc:330] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401: stopping tablet replica
I20260620 02:58:58.356611 2642 tablet_bootstrap.cc:654] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:58.356865 2693 raft_consensus.cc:2229] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:58:58.357596 2693 raft_consensus.cc:2258] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:58:58.359578 2693 ts_tablet_manager.cc:1905] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:58:58.360165 2691 tablet_replica.cc:330] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15: stopping tablet replica
I20260620 02:58:58.367606 2693 ts_tablet_manager.cc:1918] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:58:58.368728 2691 raft_consensus.cc:2229] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:58:58.369170 2691 raft_consensus.cc:2258] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:58:58.371129 2691 ts_tablet_manager.cc:1905] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:58:58.372026 2641 tablet_bootstrap.cc:492] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4: Bootstrap starting.
I20260620 02:58:58.373977 2641 tablet_bootstrap.cc:654] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:58.375540 2642 tablet_bootstrap.cc:492] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15: No bootstrap required, opened a new log
I20260620 02:58:58.375743 2642 ts_tablet_manager.cc:1397] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.009s
I20260620 02:58:58.376797 2642 raft_consensus.cc:348] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:58:58.377125 2642 raft_consensus.cc:374] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:58.370049 2640 tablet_bootstrap.cc:492] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401: Bootstrap starting.
I20260620 02:58:58.377209 2642 raft_consensus.cc:729] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a13dae5378be431eaacbde2403e5be15, State: Initialized, Role: FOLLOWER
I20260620 02:58:58.377548 2642 consensus_queue.cc:260] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:58:58.378847 2640 tablet_bootstrap.cc:654] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:58.379279 2642 ts_tablet_manager.cc:1428] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 02:58:58.383734 2693 log.cc:1196] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/98503fc8966b48d097387ff66b118af4
I20260620 02:58:58.384577 2693 ts_tablet_manager.cc:1939] T 98503fc8966b48d097387ff66b118af4 P f4b895774d8345309ddebd223feb4401: Deleting consensus metadata
I20260620 02:58:58.386983 2121 catalog_manager.cc:4841] TS f4b895774d8345309ddebd223feb4401 (127.0.97.2:32897): tablet 98503fc8966b48d097387ff66b118af4 (table test-table [id=1a2e1e83a4bb46ef89f0359f12906803]) successfully deleted
I20260620 02:58:58.389500 2691 ts_tablet_manager.cc:1918] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:58:58.392127 2691 log.cc:1196] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/98503fc8966b48d097387ff66b118af4
I20260620 02:58:58.392890 2691 ts_tablet_manager.cc:1939] T 98503fc8966b48d097387ff66b118af4 P a13dae5378be431eaacbde2403e5be15: Deleting consensus metadata
I20260620 02:58:58.396155 2123 catalog_manager.cc:4841] TS a13dae5378be431eaacbde2403e5be15 (127.0.97.3:34871): tablet 98503fc8966b48d097387ff66b118af4 (table test-table [id=1a2e1e83a4bb46ef89f0359f12906803]) successfully deleted
I20260620 02:58:58.397780 2640 tablet_bootstrap.cc:492] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401: No bootstrap required, opened a new log
I20260620 02:58:58.397796 2641 tablet_bootstrap.cc:492] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4: No bootstrap required, opened a new log
I20260620 02:58:58.397975 2641 ts_tablet_manager.cc:1397] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.004s
I20260620 02:58:58.398073 2640 ts_tablet_manager.cc:1397] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.005s
I20260620 02:58:58.398798 2641 raft_consensus.cc:348] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:58:58.399066 2641 raft_consensus.cc:374] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:58.399150 2641 raft_consensus.cc:729] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 561036292d05476c840ea0e9636a1bf4, State: Initialized, Role: FOLLOWER
I20260620 02:58:58.398914 2640 raft_consensus.cc:348] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:58:58.399528 2640 raft_consensus.cc:374] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:58:58.399408 2641 consensus_queue.cc:260] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:58:58.399682 2640 raft_consensus.cc:729] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b895774d8345309ddebd223feb4401, State: Initialized, Role: FOLLOWER
I20260620 02:58:58.399995 2640 consensus_queue.cc:260] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:58:58.402557 2641 ts_tablet_manager.cc:1428] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 02:58:58.403493 2692 tablet_replica.cc:330] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4: stopping tablet replica
I20260620 02:58:58.403947 2692 raft_consensus.cc:2229] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:58:58.404289 2692 raft_consensus.cc:2258] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:58:58.405822 2640 ts_tablet_manager.cc:1428] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260620 02:58:58.406888 2692 ts_tablet_manager.cc:1905] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:58:58.414884 2692 ts_tablet_manager.cc:1918] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:58:58.415064 2692 log.cc:1196] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/98503fc8966b48d097387ff66b118af4
I20260620 02:58:58.415594 2692 ts_tablet_manager.cc:1939] T 98503fc8966b48d097387ff66b118af4 P 561036292d05476c840ea0e9636a1bf4: Deleting consensus metadata
I20260620 02:58:58.417344 2124 catalog_manager.cc:4841] TS 561036292d05476c840ea0e9636a1bf4 (127.0.97.1:38291): tablet 98503fc8966b48d097387ff66b118af4 (table test-table [id=1a2e1e83a4bb46ef89f0359f12906803]) successfully deleted
I20260620 02:58:58.499104 2688 raft_consensus.cc:482] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:58:58.499353 2688 raft_consensus.cc:504] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:58:58.500267 2688 leader_election.cc:288] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4b895774d8345309ddebd223feb4401 (127.0.97.2:32897), a13dae5378be431eaacbde2403e5be15 (127.0.97.3:34871)
I20260620 02:58:58.500828 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b091fd521c4d416d87cedd9851f953d5" candidate_uuid: "561036292d05476c840ea0e9636a1bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b895774d8345309ddebd223feb4401" is_pre_election: true
I20260620 02:58:58.501068 2432 raft_consensus.cc:2454] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 561036292d05476c840ea0e9636a1bf4 in term 0.
I20260620 02:58:58.501017 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b091fd521c4d416d87cedd9851f953d5" candidate_uuid: "561036292d05476c840ea0e9636a1bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a13dae5378be431eaacbde2403e5be15" is_pre_election: true
I20260620 02:58:58.501277 2576 raft_consensus.cc:2454] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 561036292d05476c840ea0e9636a1bf4 in term 0.
I20260620 02:58:58.501793 2224 leader_election.cc:302] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 561036292d05476c840ea0e9636a1bf4, f4b895774d8345309ddebd223feb4401; no voters:
I20260620 02:58:58.502321 2688 raft_consensus.cc:2781] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:58:58.502439 2688 raft_consensus.cc:482] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:58:58.502539 2688 raft_consensus.cc:3037] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:58.505934 2688 raft_consensus.cc:504] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:58:58.506759 2688 leader_election.cc:288] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [CANDIDATE]: Term 1 election: Requested vote from peers f4b895774d8345309ddebd223feb4401 (127.0.97.2:32897), a13dae5378be431eaacbde2403e5be15 (127.0.97.3:34871)
I20260620 02:58:58.507118 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b091fd521c4d416d87cedd9851f953d5" candidate_uuid: "561036292d05476c840ea0e9636a1bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a13dae5378be431eaacbde2403e5be15"
I20260620 02:58:58.507325 2576 raft_consensus.cc:3037] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:58.507279 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b091fd521c4d416d87cedd9851f953d5" candidate_uuid: "561036292d05476c840ea0e9636a1bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b895774d8345309ddebd223feb4401"
I20260620 02:58:58.507479 2432 raft_consensus.cc:3037] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:58.510309 2576 raft_consensus.cc:2454] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 561036292d05476c840ea0e9636a1bf4 in term 1.
I20260620 02:58:58.510540 2432 raft_consensus.cc:2454] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 561036292d05476c840ea0e9636a1bf4 in term 1.
I20260620 02:58:58.510761 2226 leader_election.cc:302] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 561036292d05476c840ea0e9636a1bf4, a13dae5378be431eaacbde2403e5be15; no voters:
I20260620 02:58:58.511162 2688 raft_consensus.cc:2781] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:58:58.511430 2688 raft_consensus.cc:686] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 561036292d05476c840ea0e9636a1bf4, State: Running, Role: LEADER
I20260620 02:58:58.511727 2688 consensus_queue.cc:237] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:58:58.519572 2137 catalog_manager.cc:5495] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 561036292d05476c840ea0e9636a1bf4 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "561036292d05476c840ea0e9636a1bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } health_report { overall_health: UNKNOWN } } }
I20260620 02:58:58.546917 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:58.540193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39068 (local address 127.0.97.1:38291)
0620 02:58:58.540398 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:58.540404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:58.540672 (+ 268us) server_negotiation.cc:407] Connection header received
0620 02:58:58.540860 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:58.540865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:58.540951 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:58.541051 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:58:58.541442 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.542767 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.544754 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.545490 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.545694 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:58.546104 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:58:58.546264 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:58:58.546345 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:58.546518 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 02:58:58.546600 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 02:58:58.562697 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:58.556072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38291 (local address 127.0.97.1:59559)
0620 02:58:58.556392 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:58.556413 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:58.556518 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:58.556991 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:58.556995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:58.557006 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:58:58.557194 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:58.557204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.558702 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.558708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:58.560432 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:58.560446 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.561416 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.561421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:58.561629 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:58.561678 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:58:58.562150 (+ 472us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:58:58.562161 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:58:58.562229 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 02:58:58.562365 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260620 02:58:58.563385 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:58.556068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59559 (local address 127.0.97.1:38291)
0620 02:58:58.556729 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:58.556735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:58.556749 (+ 14us) server_negotiation.cc:407] Connection header received
0620 02:58:58.556795 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:58.556802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:58.556868 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:58.557305 (+ 437us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:58:58.557411 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.558560 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.560607 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:58.561258 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:58.561807 (+ 549us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:58.561910 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:58:58.562046 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:58:58.563014 (+ 968us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:58.563061 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 02:58:58.563137 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534}
I20260620 02:58:58.574342 2432 raft_consensus.cc:1261] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401 [term 1 FOLLOWER]: Refusing update from remote peer 561036292d05476c840ea0e9636a1bf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:58:58.574383 2576 raft_consensus.cc:1261] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15 [term 1 FOLLOWER]: Refusing update from remote peer 561036292d05476c840ea0e9636a1bf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:58:58.574929 2664 consensus_queue.cc:1035] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:58:58.575333 2688 consensus_queue.cc:1035] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:58:58.614634 2669 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 02:59:00.214910 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:00.215193 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:00.219306 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045: Sending DeleteTablet for 3 replicas of tablet b091fd521c4d416d87cedd9851f953d5
I20260620 02:59:00.219973 2273 tablet_service.cc:1514] Processing DeleteTablet for tablet b091fd521c4d416d87cedd9851f953d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39046
I20260620 02:59:00.220126 2411 tablet_service.cc:1514] Processing DeleteTablet for tablet b091fd521c4d416d87cedd9851f953d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49138
I20260620 02:59:00.220191 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet b091fd521c4d416d87cedd9851f953d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43476
I20260620 02:59:00.231580 2721 tablet_replica.cc:330] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401: stopping tablet replica
I20260620 02:59:00.231750 2720 tablet_replica.cc:330] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4: stopping tablet replica
I20260620 02:59:00.231928 2721 raft_consensus.cc:2229] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:00.231981 2720 raft_consensus.cc:2229] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:59:00.232254 2721 raft_consensus.cc:2258] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:00.232561 2720 raft_consensus.cc:2258] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:00.232970 2722 tablet_replica.cc:330] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15: stopping tablet replica
I20260620 02:59:00.233199 2722 raft_consensus.cc:2229] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:00.233259 2721 ts_tablet_manager.cc:1905] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:00.233505 2722 raft_consensus.cc:2258] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:00.233547 2720 ts_tablet_manager.cc:1905] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:00.234501 2722 ts_tablet_manager.cc:1905] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:00.240077 2721 ts_tablet_manager.cc:1918] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:59:00.240235 2721 log.cc:1196] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/b091fd521c4d416d87cedd9851f953d5
I20260620 02:59:00.240656 2721 ts_tablet_manager.cc:1939] T b091fd521c4d416d87cedd9851f953d5 P f4b895774d8345309ddebd223feb4401: Deleting consensus metadata
I20260620 02:59:00.241151 2722 ts_tablet_manager.cc:1918] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:59:00.241257 2720 ts_tablet_manager.cc:1918] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:59:00.241420 2720 log.cc:1196] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/b091fd521c4d416d87cedd9851f953d5
I20260620 02:59:00.241384 2722 log.cc:1196] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/b091fd521c4d416d87cedd9851f953d5
I20260620 02:59:00.241988 2720 ts_tablet_manager.cc:1939] T b091fd521c4d416d87cedd9851f953d5 P 561036292d05476c840ea0e9636a1bf4: Deleting consensus metadata
I20260620 02:59:00.241943 2121 catalog_manager.cc:4841] TS f4b895774d8345309ddebd223feb4401 (127.0.97.2:32897): tablet b091fd521c4d416d87cedd9851f953d5 (table test-table [id=3a1e58582aae456ab9ad0667e2b99504]) successfully deleted
I20260620 02:59:00.242004 2722 ts_tablet_manager.cc:1939] T b091fd521c4d416d87cedd9851f953d5 P a13dae5378be431eaacbde2403e5be15: Deleting consensus metadata
I20260620 02:59:00.243152 2124 catalog_manager.cc:4841] TS 561036292d05476c840ea0e9636a1bf4 (127.0.97.1:38291): tablet b091fd521c4d416d87cedd9851f953d5 (table test-table [id=3a1e58582aae456ab9ad0667e2b99504]) successfully deleted
I20260620 02:59:00.243220 2123 catalog_manager.cc:4841] TS a13dae5378be431eaacbde2403e5be15 (127.0.97.3:34871): tablet b091fd521c4d416d87cedd9851f953d5 (table test-table [id=3a1e58582aae456ab9ad0667e2b99504]) successfully deleted
Jun 20 02:59:00 dist-test-slave-f1j0 krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924340, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:59:00 dist-test-slave-f1j0 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924340, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:00.278860 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:00.263513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41274 (local address 127.0.97.62:40285)
0620 02:59:00.264129 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:00.264137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:00.264189 (+ 52us) server_negotiation.cc:407] Connection header received
0620 02:59:00.264423 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:00.264434 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:00.264498 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:00.264623 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:00.266033 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.266792 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.268066 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.268512 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.268664 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:00.273269 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:00.273275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:00.273278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:00.273314 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:00.276097 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:00.276796 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:00.276802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:00.276806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:00.276925 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:00.277302 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:00.277309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:00.277312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:00.277690 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:00.277921 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:00.278258 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 02:59:00.278478 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":316,"threads_started":1}
I20260620 02:59:00.282644 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:59:00.283299 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:59:00.289446 2273 tablet_service.cc:1467] Processing CreateTablet for tablet f4895424f44744a0a54af8f07552847e (DEFAULT_TABLE table=test-table [id=1c89dc931e1a4753ac5fbb819572a1a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:00.289695 2411 tablet_service.cc:1467] Processing CreateTablet for tablet f4895424f44744a0a54af8f07552847e (DEFAULT_TABLE table=test-table [id=1c89dc931e1a4753ac5fbb819572a1a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:00.289919 2273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4895424f44744a0a54af8f07552847e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:00.290114 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4895424f44744a0a54af8f07552847e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:00.290032 2550 tablet_service.cc:1467] Processing CreateTablet for tablet f4895424f44744a0a54af8f07552847e (DEFAULT_TABLE table=test-table [id=1c89dc931e1a4753ac5fbb819572a1a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:00.290445 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4895424f44744a0a54af8f07552847e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:00.296964 2732 tablet_bootstrap.cc:492] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401: Bootstrap starting.
I20260620 02:59:00.298619 2731 tablet_bootstrap.cc:492] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4: Bootstrap starting.
I20260620 02:59:00.299793 2732 tablet_bootstrap.cc:654] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:00.300813 2731 tablet_bootstrap.cc:654] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:00.301545 2732 tablet_bootstrap.cc:492] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401: No bootstrap required, opened a new log
I20260620 02:59:00.301839 2732 ts_tablet_manager.cc:1397] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 02:59:00.302086 2733 tablet_bootstrap.cc:492] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15: Bootstrap starting.
I20260620 02:59:00.302839 2732 raft_consensus.cc:348] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:59:00.303182 2732 raft_consensus.cc:374] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:00.303314 2732 raft_consensus.cc:729] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b895774d8345309ddebd223feb4401, State: Initialized, Role: FOLLOWER
I20260620 02:59:00.303735 2733 tablet_bootstrap.cc:654] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:00.303627 2732 consensus_queue.cc:260] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:59:00.304953 2732 ts_tablet_manager.cc:1428] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 02:59:00.306692 2733 tablet_bootstrap.cc:492] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15: No bootstrap required, opened a new log
I20260620 02:59:00.306859 2733 ts_tablet_manager.cc:1397] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 02:59:00.306905 2731 tablet_bootstrap.cc:492] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4: No bootstrap required, opened a new log
I20260620 02:59:00.307067 2731 ts_tablet_manager.cc:1397] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 02:59:00.307766 2731 raft_consensus.cc:348] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:59:00.308012 2731 raft_consensus.cc:374] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:00.308109 2731 raft_consensus.cc:729] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 561036292d05476c840ea0e9636a1bf4, State: Initialized, Role: FOLLOWER
I20260620 02:59:00.308073 2733 raft_consensus.cc:348] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:59:00.308383 2733 raft_consensus.cc:374] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:00.308342 2731 consensus_queue.cc:260] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:59:00.308480 2733 raft_consensus.cc:729] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a13dae5378be431eaacbde2403e5be15, State: Initialized, Role: FOLLOWER
I20260620 02:59:00.308915 2731 ts_tablet_manager.cc:1428] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 02:59:00.308885 2733 consensus_queue.cc:260] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:59:00.310086 2733 ts_tablet_manager.cc:1428] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 02:59:00.518949 2738 raft_consensus.cc:482] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:59:00.519177 2738 raft_consensus.cc:504] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:59:00.520871 2738 leader_election.cc:288] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 561036292d05476c840ea0e9636a1bf4 (127.0.97.1:38291), f4b895774d8345309ddebd223feb4401 (127.0.97.2:32897)
I20260620 02:59:00.542858 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:00.521431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49973 (local address 127.0.97.2:32897)
0620 02:59:00.535349 (+ 13918us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:00.535354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:00.535375 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:59:00.535459 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:00.535466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:00.535559 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:00.535657 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:00.536379 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.537558 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.540628 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.541865 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.542011 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:00.542143 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:00.542339 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 02:59:00.542471 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13769,"thread_start_us":13497,"threads_started":1}
I20260620 02:59:00.542845 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:00.521337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32897 (local address 127.0.97.3:49973)
0620 02:59:00.534642 (+ 13305us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:00.534670 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:00.534850 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:00.535674 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:00.535680 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:00.535713 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:00.536176 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:00.536183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.537730 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.537736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:00.540440 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:00.540454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.542069 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.542074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:00.542183 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:00.542208 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 02:59:00.542292 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 02:59:00.542425 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13142,"mutex_wait_us":241,"spinlock_wait_cycles":896,"thread_start_us":12537,"threads_started":1}
I20260620 02:59:00.543424 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:00.521175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50379 (local address 127.0.97.1:38291)
0620 02:59:00.534966 (+ 13791us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:00.534973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:00.535236 (+ 263us) server_negotiation.cc:407] Connection header received
0620 02:59:00.535332 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:00.535338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:00.535423 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:00.535562 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:00.536379 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.537608 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.540627 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.541907 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.542039 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:00.542143 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:00.542947 (+ 804us) server_negotiation.cc:299] Negotiation successful
0620 02:59:00.543110 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13624,"thread_start_us":13475,"threads_started":1}
I20260620 02:59:00.543869 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:00.521131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38291 (local address 127.0.97.3:50379)
0620 02:59:00.535076 (+ 13945us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:00.535091 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:00.535179 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:00.535613 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:00.535620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:00.535650 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:00.536168 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:00.536180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.537732 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.537736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:00.540440 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:00.540454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.542313 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.542319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:00.542485 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:00.542522 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:59:00.542613 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 02:59:00.542742 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12922,"mutex_wait_us":265,"spinlock_wait_cycles":7296,"thread_start_us":13147,"threads_started":1}
I20260620 02:59:00.544003 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4895424f44744a0a54af8f07552847e" candidate_uuid: "a13dae5378be431eaacbde2403e5be15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b895774d8345309ddebd223feb4401" is_pre_election: true
I20260620 02:59:00.544309 2432 raft_consensus.cc:2454] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a13dae5378be431eaacbde2403e5be15 in term 0.
I20260620 02:59:00.544539 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4895424f44744a0a54af8f07552847e" candidate_uuid: "a13dae5378be431eaacbde2403e5be15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "561036292d05476c840ea0e9636a1bf4" is_pre_election: true
I20260620 02:59:00.544823 2298 raft_consensus.cc:2454] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a13dae5378be431eaacbde2403e5be15 in term 0.
I20260620 02:59:00.544885 2502 leader_election.cc:302] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a13dae5378be431eaacbde2403e5be15, f4b895774d8345309ddebd223feb4401; no voters:
I20260620 02:59:00.545349 2738 raft_consensus.cc:2781] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:59:00.545485 2738 raft_consensus.cc:482] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:59:00.545614 2738 raft_consensus.cc:3037] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:00.548098 2738 raft_consensus.cc:504] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:59:00.548693 2738 leader_election.cc:288] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [CANDIDATE]: Term 1 election: Requested vote from peers 561036292d05476c840ea0e9636a1bf4 (127.0.97.1:38291), f4b895774d8345309ddebd223feb4401 (127.0.97.2:32897)
I20260620 02:59:00.549137 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4895424f44744a0a54af8f07552847e" candidate_uuid: "a13dae5378be431eaacbde2403e5be15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "561036292d05476c840ea0e9636a1bf4"
I20260620 02:59:00.549187 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4895424f44744a0a54af8f07552847e" candidate_uuid: "a13dae5378be431eaacbde2403e5be15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b895774d8345309ddebd223feb4401"
I20260620 02:59:00.549350 2298 raft_consensus.cc:3037] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:00.549384 2432 raft_consensus.cc:3037] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:00.552299 2298 raft_consensus.cc:2454] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a13dae5378be431eaacbde2403e5be15 in term 1.
I20260620 02:59:00.552299 2432 raft_consensus.cc:2454] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a13dae5378be431eaacbde2403e5be15 in term 1.
I20260620 02:59:00.552765 2505 leader_election.cc:302] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 561036292d05476c840ea0e9636a1bf4, a13dae5378be431eaacbde2403e5be15; no voters:
I20260620 02:59:00.553150 2738 raft_consensus.cc:2781] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:00.553547 2738 raft_consensus.cc:686] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [term 1 LEADER]: Becoming Leader. State: Replica: a13dae5378be431eaacbde2403e5be15, State: Running, Role: LEADER
I20260620 02:59:00.553977 2738 consensus_queue.cc:237] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:59:00.557505 2136 catalog_manager.cc:5495] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 reported cstate change: term changed from 0 to 1, leader changed from <none> to a13dae5378be431eaacbde2403e5be15 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "a13dae5378be431eaacbde2403e5be15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } health_report { overall_health: HEALTHY } } }
I20260620 02:59:00.603524 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:00.597143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.0.97.3:34871)
0620 02:59:00.597727 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:00.597731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:00.597748 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:59:00.597827 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:00.597831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:00.597898 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:00.598097 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:59:00.598537 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.599387 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.600739 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.601156 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.601311 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:00.602252 (+ 941us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:59:00.602851 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:59:00.603046 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:00.603106 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 02:59:00.603184 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":292,"threads_started":1}
I20260620 02:59:00.620049 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:00.613651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59093 (local address 127.0.97.1:38291)
0620 02:59:00.613836 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:00.613842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:00.614544 (+ 702us) server_negotiation.cc:407] Connection header received
0620 02:59:00.614698 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:00.614702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:00.614772 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:00.614862 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:59:00.615542 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.616400 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.618485 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.618873 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.618968 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:00.619495 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:59:00.619589 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:59:00.619659 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:00.619801 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 02:59:00.619859 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:59:00.620745 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:00.613463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38291 (local address 127.0.97.3:59093)
0620 02:59:00.614399 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:00.614418 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:00.614562 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:00.614895 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:00.614900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:00.614912 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:59:00.615337 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:00.615348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.616531 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.616537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:00.618345 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:00.618359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.619056 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.619062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:00.619258 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:00.619326 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:59:00.619686 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:59:00.619697 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:59:00.619771 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 02:59:00.619909 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":570,"threads_started":1}
I20260620 02:59:00.631747 2432 raft_consensus.cc:1261] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401 [term 1 FOLLOWER]: Refusing update from remote peer a13dae5378be431eaacbde2403e5be15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:00.632236 2298 raft_consensus.cc:1261] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4 [term 1 FOLLOWER]: Refusing update from remote peer a13dae5378be431eaacbde2403e5be15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:00.633052 2743 consensus_queue.cc:1035] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:00.633733 2738 consensus_queue.cc:1035] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:00.668826 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:00.663009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33169 (local address 127.0.97.3:34871)
0620 02:59:00.663196 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:00.663204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:00.663467 (+ 263us) server_negotiation.cc:407] Connection header received
0620 02:59:00.663674 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:00.663679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:00.663750 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:00.663846 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:59:00.664784 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.665491 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.667166 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.667563 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.667712 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:00.668098 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:59:00.668195 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:59:00.668299 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:00.668434 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 02:59:00.668529 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 02:59:00.668880 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:00.662717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34871 (local address 127.0.97.1:33169)
0620 02:59:00.663359 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:00.663382 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:00.663537 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:00.663860 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:00.663865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:00.663876 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:59:00.664569 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:00.664577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.665661 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.665667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:00.667017 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:00.667028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:00.667744 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:00.667750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:00.667880 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:00.667914 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:59:00.668301 (+ 387us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:59:00.668313 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 02:59:00.668414 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 02:59:00.668541 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":511,"threads_started":1}
I20260620 02:59:00.703374 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:00.703647 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:00.706813 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045: Sending DeleteTablet for 3 replicas of tablet f4895424f44744a0a54af8f07552847e
I20260620 02:59:00.707741 2273 tablet_service.cc:1514] Processing DeleteTablet for tablet f4895424f44744a0a54af8f07552847e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39046
I20260620 02:59:00.707842 2411 tablet_service.cc:1514] Processing DeleteTablet for tablet f4895424f44744a0a54af8f07552847e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49138
I20260620 02:59:00.708103 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet f4895424f44744a0a54af8f07552847e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43476
I20260620 02:59:00.708159 2720 tablet_replica.cc:330] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4: stopping tablet replica
I20260620 02:59:00.708227 2721 tablet_replica.cc:330] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401: stopping tablet replica
I20260620 02:59:00.708369 2720 raft_consensus.cc:2229] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:00.708427 2721 raft_consensus.cc:2229] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:00.708606 2721 raft_consensus.cc:2258] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:00.708600 2720 raft_consensus.cc:2258] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:00.708894 2722 tablet_replica.cc:330] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15: stopping tablet replica
I20260620 02:59:00.709120 2722 raft_consensus.cc:2229] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:59:00.709771 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 02:59:00.710198 2721 ts_tablet_manager.cc:1905] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 02:59:00.710445 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:59:00.710877 2720 ts_tablet_manager.cc:1905] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:00.713547 2722 raft_consensus.cc:2258] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:00.714712 2722 ts_tablet_manager.cc:1905] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:00.717792 2721 ts_tablet_manager.cc:1918] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:59:00.717947 2721 log.cc:1196] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/f4895424f44744a0a54af8f07552847e
I20260620 02:59:00.718405 2721 ts_tablet_manager.cc:1939] T f4895424f44744a0a54af8f07552847e P f4b895774d8345309ddebd223feb4401: Deleting consensus metadata
I20260620 02:59:00.719987 2121 catalog_manager.cc:4841] TS f4b895774d8345309ddebd223feb4401 (127.0.97.2:32897): tablet f4895424f44744a0a54af8f07552847e (table test-table [id=1c89dc931e1a4753ac5fbb819572a1a5]) successfully deleted
I20260620 02:59:00.721467 2722 ts_tablet_manager.cc:1918] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:59:00.721824 2722 log.cc:1196] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/f4895424f44744a0a54af8f07552847e
I20260620 02:59:00.722365 2273 tablet_service.cc:1467] Processing CreateTablet for tablet 0e16162c2bd1496482e08b8c48840089 (DEFAULT_TABLE table=test-table [id=bf21ed79329744eb9f79acd7e1c2f16b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:00.722648 2550 tablet_service.cc:1467] Processing CreateTablet for tablet 0e16162c2bd1496482e08b8c48840089 (DEFAULT_TABLE table=test-table [id=bf21ed79329744eb9f79acd7e1c2f16b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:00.722834 2273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e16162c2bd1496482e08b8c48840089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:00.722779 2411 tablet_service.cc:1467] Processing CreateTablet for tablet 0e16162c2bd1496482e08b8c48840089 (DEFAULT_TABLE table=test-table [id=bf21ed79329744eb9f79acd7e1c2f16b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:00.723102 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e16162c2bd1496482e08b8c48840089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:00.723177 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e16162c2bd1496482e08b8c48840089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:00.730130 2731 tablet_bootstrap.cc:492] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4: Bootstrap starting.
I20260620 02:59:00.730175 2733 tablet_bootstrap.cc:492] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15: Bootstrap starting.
I20260620 02:59:00.731860 2731 tablet_bootstrap.cc:654] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:00.732259 2733 tablet_bootstrap.cc:654] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:00.733943 2720 ts_tablet_manager.cc:1918] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:59:00.734174 2720 log.cc:1196] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/f4895424f44744a0a54af8f07552847e
I20260620 02:59:00.734759 2720 ts_tablet_manager.cc:1939] T f4895424f44744a0a54af8f07552847e P 561036292d05476c840ea0e9636a1bf4: Deleting consensus metadata
I20260620 02:59:00.735069 2732 tablet_bootstrap.cc:492] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401: Bootstrap starting.
I20260620 02:59:00.722332 2722 ts_tablet_manager.cc:1939] T f4895424f44744a0a54af8f07552847e P a13dae5378be431eaacbde2403e5be15: Deleting consensus metadata
I20260620 02:59:00.737073 2732 tablet_bootstrap.cc:654] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:00.737483 2124 catalog_manager.cc:4841] TS 561036292d05476c840ea0e9636a1bf4 (127.0.97.1:38291): tablet f4895424f44744a0a54af8f07552847e (table test-table [id=1c89dc931e1a4753ac5fbb819572a1a5]) successfully deleted
I20260620 02:59:00.738652 2123 catalog_manager.cc:4841] TS a13dae5378be431eaacbde2403e5be15 (127.0.97.3:34871): tablet f4895424f44744a0a54af8f07552847e (table test-table [id=1c89dc931e1a4753ac5fbb819572a1a5]) successfully deleted
I20260620 02:59:00.739583 2731 tablet_bootstrap.cc:492] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4: No bootstrap required, opened a new log
I20260620 02:59:00.739751 2731 ts_tablet_manager.cc:1397] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 02:59:00.740613 2731 raft_consensus.cc:348] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:59:00.740857 2731 raft_consensus.cc:374] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:00.740940 2731 raft_consensus.cc:729] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 561036292d05476c840ea0e9636a1bf4, State: Initialized, Role: FOLLOWER
I20260620 02:59:00.741205 2731 consensus_queue.cc:260] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:59:00.741786 2731 ts_tablet_manager.cc:1428] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 02:59:00.743575 2733 tablet_bootstrap.cc:492] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15: No bootstrap required, opened a new log
I20260620 02:59:00.743824 2733 ts_tablet_manager.cc:1397] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260620 02:59:00.744740 2732 tablet_bootstrap.cc:492] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401: No bootstrap required, opened a new log
I20260620 02:59:00.744907 2732 ts_tablet_manager.cc:1397] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 02:59:00.744766 2733 raft_consensus.cc:348] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:59:00.745214 2733 raft_consensus.cc:374] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:00.745725 2732 raft_consensus.cc:348] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:59:00.745959 2732 raft_consensus.cc:374] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:00.745891 2733 raft_consensus.cc:729] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a13dae5378be431eaacbde2403e5be15, State: Initialized, Role: FOLLOWER
I20260620 02:59:00.746052 2732 raft_consensus.cc:729] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b895774d8345309ddebd223feb4401, State: Initialized, Role: FOLLOWER
I20260620 02:59:00.746366 2732 consensus_queue.cc:260] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:59:00.746399 2733 consensus_queue.cc:260] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:59:00.747146 2733 ts_tablet_manager.cc:1428] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 02:59:00.747773 2732 ts_tablet_manager.cc:1428] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 02:59:00.973856 2743 raft_consensus.cc:482] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:59:00.974162 2743 raft_consensus.cc:504] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:59:00.974974 2743 leader_election.cc:288] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 561036292d05476c840ea0e9636a1bf4 (127.0.97.1:38291), f4b895774d8345309ddebd223feb4401 (127.0.97.2:32897)
I20260620 02:59:00.975525 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e16162c2bd1496482e08b8c48840089" candidate_uuid: "a13dae5378be431eaacbde2403e5be15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "561036292d05476c840ea0e9636a1bf4" is_pre_election: true
I20260620 02:59:00.975577 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e16162c2bd1496482e08b8c48840089" candidate_uuid: "a13dae5378be431eaacbde2403e5be15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b895774d8345309ddebd223feb4401" is_pre_election: true
I20260620 02:59:00.975785 2298 raft_consensus.cc:2454] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a13dae5378be431eaacbde2403e5be15 in term 0.
I20260620 02:59:00.975803 2431 raft_consensus.cc:2454] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a13dae5378be431eaacbde2403e5be15 in term 0.
I20260620 02:59:00.976248 2502 leader_election.cc:302] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a13dae5378be431eaacbde2403e5be15, f4b895774d8345309ddebd223feb4401; no voters:
I20260620 02:59:00.976575 2743 raft_consensus.cc:2781] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:59:00.976689 2743 raft_consensus.cc:482] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:59:00.976800 2743 raft_consensus.cc:3037] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:00.980031 2743 raft_consensus.cc:504] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:59:00.980684 2743 leader_election.cc:288] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [CANDIDATE]: Term 1 election: Requested vote from peers 561036292d05476c840ea0e9636a1bf4 (127.0.97.1:38291), f4b895774d8345309ddebd223feb4401 (127.0.97.2:32897)
I20260620 02:59:00.981185 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e16162c2bd1496482e08b8c48840089" candidate_uuid: "a13dae5378be431eaacbde2403e5be15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b895774d8345309ddebd223feb4401"
I20260620 02:59:00.981197 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e16162c2bd1496482e08b8c48840089" candidate_uuid: "a13dae5378be431eaacbde2403e5be15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "561036292d05476c840ea0e9636a1bf4"
I20260620 02:59:00.981397 2431 raft_consensus.cc:3037] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:00.981411 2298 raft_consensus.cc:3037] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:00.983752 2431 raft_consensus.cc:2454] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a13dae5378be431eaacbde2403e5be15 in term 1.
I20260620 02:59:00.984319 2502 leader_election.cc:302] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a13dae5378be431eaacbde2403e5be15, f4b895774d8345309ddebd223feb4401; no voters:
I20260620 02:59:00.984628 2743 raft_consensus.cc:2781] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:00.984813 2743 raft_consensus.cc:686] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [term 1 LEADER]: Becoming Leader. State: Replica: a13dae5378be431eaacbde2403e5be15, State: Running, Role: LEADER
I20260620 02:59:00.984817 2298 raft_consensus.cc:2454] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a13dae5378be431eaacbde2403e5be15 in term 1.
I20260620 02:59:00.985085 2743 consensus_queue.cc:237] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } }
I20260620 02:59:00.988180 2137 catalog_manager.cc:5495] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 reported cstate change: term changed from 0 to 1, leader changed from <none> to a13dae5378be431eaacbde2403e5be15 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "a13dae5378be431eaacbde2403e5be15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a13dae5378be431eaacbde2403e5be15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34871 } health_report { overall_health: HEALTHY } } }
I20260620 02:59:01.051769 2298 raft_consensus.cc:1261] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4 [term 1 FOLLOWER]: Refusing update from remote peer a13dae5378be431eaacbde2403e5be15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:01.052539 2738 consensus_queue.cc:1035] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "561036292d05476c840ea0e9636a1bf4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:01.058629 2431 raft_consensus.cc:1261] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401 [term 1 FOLLOWER]: Refusing update from remote peer a13dae5378be431eaacbde2403e5be15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:01.059444 2743 consensus_queue.cc:1035] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b895774d8345309ddebd223feb4401" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:59:01.088279 2704 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 02:59:02.697098 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:02.697299 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:02.701725 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2db0c28a722c4a6fbf20e9eb269bf045: Sending DeleteTablet for 3 replicas of tablet 0e16162c2bd1496482e08b8c48840089
I20260620 02:59:02.702576 2272 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e16162c2bd1496482e08b8c48840089 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39046
I20260620 02:59:02.702764 2411 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e16162c2bd1496482e08b8c48840089 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49138
I20260620 02:59:02.702786 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e16162c2bd1496482e08b8c48840089 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43476
I20260620 02:59:02.703691 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 2208
I20260620 02:59:02.706717 2791 tablet_replica.cc:330] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15: stopping tablet replica
I20260620 02:59:02.706871 2789 tablet_replica.cc:330] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4: stopping tablet replica
I20260620 02:59:02.706926 2791 raft_consensus.cc:2229] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:59:02.707038 2789 raft_consensus.cc:2229] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:02.707257 2789 raft_consensus.cc:2258] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:02.707267 2791 raft_consensus.cc:2258] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:02.707798 2790 tablet_replica.cc:330] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401: stopping tablet replica
I20260620 02:59:02.708031 2790 raft_consensus.cc:2229] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:02.708248 2791 ts_tablet_manager.cc:1905] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:02.708284 2790 raft_consensus.cc:2258] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:02.709115 2789 ts_tablet_manager.cc:1905] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:02.709825 2790 ts_tablet_manager.cc:1905] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:02.714985 2791 ts_tablet_manager.cc:1918] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:59:02.715314 2791 log.cc:1196] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/0e16162c2bd1496482e08b8c48840089
I20260620 02:59:02.715679 2789 ts_tablet_manager.cc:1918] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:59:02.715735 2791 ts_tablet_manager.cc:1939] T 0e16162c2bd1496482e08b8c48840089 P a13dae5378be431eaacbde2403e5be15: Deleting consensus metadata
I20260620 02:59:02.715826 2789 log.cc:1196] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/0e16162c2bd1496482e08b8c48840089
I20260620 02:59:02.716398 2789 ts_tablet_manager.cc:1939] T 0e16162c2bd1496482e08b8c48840089 P 561036292d05476c840ea0e9636a1bf4: Deleting consensus metadata
I20260620 02:59:02.717020 2123 catalog_manager.cc:4841] TS a13dae5378be431eaacbde2403e5be15 (127.0.97.3:34871): tablet 0e16162c2bd1496482e08b8c48840089 (table test-table [id=bf21ed79329744eb9f79acd7e1c2f16b]) successfully deleted
I20260620 02:59:02.717314 2790 ts_tablet_manager.cc:1918] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:59:02.717450 2790 log.cc:1196] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/0e16162c2bd1496482e08b8c48840089
I20260620 02:59:02.717420 2124 catalog_manager.cc:4841] TS 561036292d05476c840ea0e9636a1bf4 (127.0.97.1:38291): tablet 0e16162c2bd1496482e08b8c48840089 (table test-table [id=bf21ed79329744eb9f79acd7e1c2f16b]) successfully deleted
I20260620 02:59:02.717903 2790 ts_tablet_manager.cc:1939] T 0e16162c2bd1496482e08b8c48840089 P f4b895774d8345309ddebd223feb4401: Deleting consensus metadata
I20260620 02:59:02.719056 2121 catalog_manager.cc:4841] TS f4b895774d8345309ddebd223feb4401 (127.0.97.2:32897): tablet 0e16162c2bd1496482e08b8c48840089 (table test-table [id=bf21ed79329744eb9f79acd7e1c2f16b]) successfully deleted
Jun 20 02:59:02 dist-test-slave-f1j0 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924340, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:59:02.725782 2793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:02.705223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.0.97.1:38291)
0620 02:59:02.708135 (+ 2912us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:02.708140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:02.708737 (+ 597us) server_negotiation.cc:407] Connection header received
0620 02:59:02.708799 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:02.708804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:02.708883 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:02.709019 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:02.710678 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:02.712217 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:02.714021 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:02.714705 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:02.714823 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:02.720720 (+ 5897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:02.720734 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:02.720750 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:02.720791 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:02.723539 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:02.724098 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:02.724104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:02.724107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:02.724230 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:02.724553 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:02.724556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:02.724558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:02.724875 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:02.725080 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:02.725309 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 02:59:02.725458 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2743,"thread_start_us":2927,"threads_started":1}
W20260620 02:59:02.726444 2338 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39084
W20260620 02:59:02.727088 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 2208: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:59:02.727289 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 2208
I20260620 02:59:02.753374 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 2347
Jun 20 02:59:02 dist-test-slave-f1j0 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924340, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:59:02.768603 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:02.754156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49154 (local address 127.0.97.2:32897)
0620 02:59:02.754690 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:02.754695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:02.754712 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:59:02.754767 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:02.754770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:02.754824 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:02.754914 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:02.756113 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:02.756858 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:02.758424 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:02.758804 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:02.758899 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:02.763093 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:02.763106 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:02.763122 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:02.763159 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:02.766240 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:02.766860 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:02.766868 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:02.766870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:02.767003 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:02.767325 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:02.767328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:02.767330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:02.767658 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:02.767873 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:02.768174 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 02:59:02.768342 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":303,"threads_started":1}
W20260620 02:59:02.769325 2477 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49154
W20260620 02:59:02.770027 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 2347: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:59:02.770206 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 2347
I20260620 02:59:02.792853 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 2486
Jun 20 02:59:02 dist-test-slave-f1j0 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924340, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:59:02.808333 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:02.793664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43494 (local address 127.0.97.3:34871)
0620 02:59:02.794270 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:02.794274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:02.794288 (+ 14us) server_negotiation.cc:407] Connection header received
0620 02:59:02.794341 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:02.794344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:02.794400 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:02.794544 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:02.795858 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:02.796614 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:02.797937 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:02.798357 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:02.798498 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:02.802782 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:02.802798 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:02.802814 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:02.802850 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:02.805423 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:02.806296 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:02.806303 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:02.806305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:02.806424 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:02.806823 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:02.806834 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:02.806837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:02.807161 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:02.807448 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:02.807773 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 02:59:02.807983 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":354,"threads_started":1}
W20260620 02:59:02.809012 2616 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43494
W20260620 02:59:02.809719 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 2486: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:59:02.809970 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 2486
I20260620 02:59:02.831348 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 2105
I20260620 02:59:02.832471 2177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:02.946218 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 2105
2026-06-20T02:59:02Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8586 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:59:03 dist-test-slave-f1j0 krb5kdc[2799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:59:03 dist-test-slave-f1j0 krb5kdc[2799](info): set up 2 sockets
Jun 20 02:59:03 dist-test-slave-f1j0 krb5kdc[2799](info): commencing operation
krb5kdc: starting...
W20260620 02:59:04.025923 388 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 02:59:04 dist-test-slave-f1j0 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:59:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:59:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:59:04.221020 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39637
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:04.538189 2812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:04.538465 2812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:04.538630 2812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:04.545710 2812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:59:04.545805 2812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:04.545892 2812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:04.545994 2812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:59:04.546053 2812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:59:04.555794 2812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:04.557191 2812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:04.558638 2812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:04.571038 2817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:04.570956 2818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:04.571007 2820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:04.571627 2812 server_base.cc:1034] running on GCE node
I20260620 02:59:04.572405 2812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:04.573803 2812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:04.575002 2812 hybrid_clock.cc:648] HybridClock initialized: now 1781924344574950 us; error 42 us; skew 500 ppm
I20260620 02:59:04.575294 2812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:04 dist-test-slave-f1j0 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:04.578318 2812 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:59:04.579717 2812 webserver.cc:460] Webserver started at http://127.0.97.62:44165/ using document root <none> and password file <none>
I20260620 02:59:04.580379 2812 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:04.580549 2812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:04.580857 2812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:04.583689 2812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "1c520e1afac245b9a2bf68c38174eb13"
format_stamp: "Formatted at 2026-06-20 02:59:04 on dist-test-slave-f1j0"
server_key: "7083b10bc1c433656170ba25fa031d5e"
server_key_iv: "c43e5f2408c5e3137d77be6f3502ef04"
server_key_version: "encryptionkey@0"
I20260620 02:59:04.584443 2812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "1c520e1afac245b9a2bf68c38174eb13"
format_stamp: "Formatted at 2026-06-20 02:59:04 on dist-test-slave-f1j0"
server_key: "7083b10bc1c433656170ba25fa031d5e"
server_key_iv: "c43e5f2408c5e3137d77be6f3502ef04"
server_key_version: "encryptionkey@0"
I20260620 02:59:04.589457 2812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 02:59:04.592979 2827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:04.594265 2812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:59:04.594462 2812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "1c520e1afac245b9a2bf68c38174eb13"
format_stamp: "Formatted at 2026-06-20 02:59:04 on dist-test-slave-f1j0"
server_key: "7083b10bc1c433656170ba25fa031d5e"
server_key_iv: "c43e5f2408c5e3137d77be6f3502ef04"
server_key_version: "encryptionkey@0"
I20260620 02:59:04.594655 2812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:04.620460 2812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:04.624401 2812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:04.624686 2812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:04.642771 2812 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39637
I20260620 02:59:04.642807 2878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39637 every 8 connection(s)
I20260620 02:59:04.644179 2812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:59:04.648036 2879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:04.648638 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 2812
I20260620 02:59:04.648867 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 02:59:04.649507 388 external_mini_cluster.cc:1428] Setting key 5aa99b21ebee194f4b5a900fd0293774
I20260620 02:59:04.660578 2879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13: Bootstrap starting.
Jun 20 02:59:04 dist-test-slave-f1j0 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:04.663940 2879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:04.665136 2879 log.cc:824] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:04.668296 2879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13: No bootstrap required, opened a new log
I20260620 02:59:04.672158 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:04.651655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44466 (local address 127.0.97.62:39637)
0620 02:59:04.652409 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:04.652420 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:04.652465 (+ 45us) server_negotiation.cc:407] Connection header received
0620 02:59:04.654470 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:04.654493 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:04.654894 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:04.655326 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:04.656749 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:04.658171 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:04.659446 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:04.660135 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:04.660342 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:04.664704 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:04.664712 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:04.664727 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:04.664768 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:04.668182 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:04.669027 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:04.669033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:04.669040 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:04.669208 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:04.669660 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:04.669664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:04.669666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:04.670158 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:04.670394 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:04.670832 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 02:59:04.671169 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":340,"threads_started":1}
I20260620 02:59:04.676927 2879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c520e1afac245b9a2bf68c38174eb13" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } }
I20260620 02:59:04.677304 2879 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:04.677439 2879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c520e1afac245b9a2bf68c38174eb13, State: Initialized, Role: FOLLOWER
I20260620 02:59:04.678040 2879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c520e1afac245b9a2bf68c38174eb13" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } }
I20260620 02:59:04.678282 2879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:59:04.678407 2879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:59:04.678579 2879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:04.680662 2879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c520e1afac245b9a2bf68c38174eb13" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } }
I20260620 02:59:04.681174 2879 leader_election.cc:302] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c520e1afac245b9a2bf68c38174eb13; no voters:
I20260620 02:59:04.681797 2879 leader_election.cc:288] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:59:04.681931 2884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:04.682698 2884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13 [term 1 LEADER]: Becoming Leader. State: Replica: 1c520e1afac245b9a2bf68c38174eb13, State: Running, Role: LEADER
I20260620 02:59:04.683285 2884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c520e1afac245b9a2bf68c38174eb13" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } }
I20260620 02:59:04.684118 2879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:59:04.686218 2886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c520e1afac245b9a2bf68c38174eb13. Latest consensus state: current_term: 1 leader_uuid: "1c520e1afac245b9a2bf68c38174eb13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c520e1afac245b9a2bf68c38174eb13" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } }
I20260620 02:59:04.686151 2885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c520e1afac245b9a2bf68c38174eb13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c520e1afac245b9a2bf68c38174eb13" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } }
I20260620 02:59:04.686569 2885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:04.686569 2886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:04.687479 2891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:59:04.692585 2891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:59:04.702191 2891 catalog_manager.cc:1347] Generated new cluster ID: 07041b5d6506462d9224dacb6b44343f
I20260620 02:59:04.702306 2891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:59:04.726256 2891 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:59:04.727571 2891 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:59:04.739167 2891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c520e1afac245b9a2bf68c38174eb13: Generated new TSK 0
I20260620 02:59:04.740262 2891 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:59:04.785435 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39637
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:05.107788 2905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:05.108100 2905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:05.108204 2905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:05.115568 2905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:05.115682 2905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:05.115796 2905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:59:05.126786 2905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:05.128612 2905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:05.130208 2905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:05.145494 2910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:05.145421 2913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:05.145469 2911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:05.146456 2905 server_base.cc:1034] running on GCE node
I20260620 02:59:05.147125 2905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:05.148103 2905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:05.149343 2905 hybrid_clock.cc:648] HybridClock initialized: now 1781924345149300 us; error 66 us; skew 500 ppm
I20260620 02:59:05.149828 2905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:05 dist-test-slave-f1j0 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:05.153012 2905 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:59:05.154451 2905 webserver.cc:460] Webserver started at http://127.0.97.1:42515/ using document root <none> and password file <none>
I20260620 02:59:05.155143 2905 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:05.155295 2905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:05.155602 2905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:05.158514 2905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "59896667f6cc4c1f9063eb57e8ce274d"
format_stamp: "Formatted at 2026-06-20 02:59:05 on dist-test-slave-f1j0"
server_key: "cf9c8b5fa9c0429f669cfc6815bf5ea9"
server_key_iv: "a16db48d0975b070407f852b2355c041"
server_key_version: "encryptionkey@0"
I20260620 02:59:05.159299 2905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "59896667f6cc4c1f9063eb57e8ce274d"
format_stamp: "Formatted at 2026-06-20 02:59:05 on dist-test-slave-f1j0"
server_key: "cf9c8b5fa9c0429f669cfc6815bf5ea9"
server_key_iv: "a16db48d0975b070407f852b2355c041"
server_key_version: "encryptionkey@0"
I20260620 02:59:05.164386 2905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 02:59:05.167834 2920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:05.169251 2905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:59:05.169425 2905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "59896667f6cc4c1f9063eb57e8ce274d"
format_stamp: "Formatted at 2026-06-20 02:59:05 on dist-test-slave-f1j0"
server_key: "cf9c8b5fa9c0429f669cfc6815bf5ea9"
server_key_iv: "a16db48d0975b070407f852b2355c041"
server_key_version: "encryptionkey@0"
I20260620 02:59:05.169667 2905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:05.210247 2905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:05.214623 2905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:05.214872 2905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:05.215739 2905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:05.216954 2905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:05.217031 2905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:05.217113 2905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:05.217155 2905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:05.249428 2905 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35553
I20260620 02:59:05.249524 3032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35553 every 8 connection(s)
I20260620 02:59:05.250767 2905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:59:05.254376 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 2905
I20260620 02:59:05.254578 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:59:05.255280 388 external_mini_cluster.cc:1428] Setting key e5b6a17583ea68b54cb6d6423f957483
Jun 20 02:59:05 dist-test-slave-f1j0 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:05.271850 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:05.253233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50405 (local address 127.0.97.62:39637)
0620 02:59:05.253694 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:05.253698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:05.254821 (+ 1123us) server_negotiation.cc:407] Connection header received
0620 02:59:05.256606 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:05.256613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:05.256693 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:05.256829 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:05.258346 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:05.259050 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:05.260572 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:05.261001 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:05.261148 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:05.264810 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:05.264819 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:05.264827 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:05.264873 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:05.268505 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:05.269243 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:05.269251 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:05.269257 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:05.269419 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:05.269912 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:05.269919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:05.269926 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:05.270342 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:05.270522 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:05.271214 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 02:59:05.271460 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":209,"threads_started":1}
I20260620 02:59:05.272645 3036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:05.253855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39637 (local address 127.0.97.1:50405)
0620 02:59:05.254632 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:05.254685 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:05.256337 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:05.257073 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:05.257082 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:05.257622 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:05.258175 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:05.258196 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:05.259218 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:05.259223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:05.260427 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:05.260437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:05.261246 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:05.261254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:05.261490 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:05.262248 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:05.262270 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:05.264578 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:05.268666 (+ 4088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:05.268674 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:05.268690 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:05.269120 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:05.269530 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:05.269533 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:05.269536 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:05.269780 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:05.270469 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:05.270475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:05.270809 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 02:59:05.271033 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 02:59:05.271442 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":343,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:59:05.274356 3033 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39637
I20260620 02:59:05.274700 3033 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:05.275451 3033 heartbeater.cc:511] Master 127.0.97.62:39637 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:59:05.277673 2844 ts_manager.cc:194] Registered new tserver with Master: 59896667f6cc4c1f9063eb57e8ce274d (127.0.97.1:35553)
I20260620 02:59:05.279125 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:59:05.300635 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39637
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:05.616722 3039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:05.617050 3039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:05.617238 3039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:05.624513 3039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:05.624670 3039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:05.624864 3039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:59:05.634405 3039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:05.635720 3039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:05.637460 3039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:05.653508 3044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:05.653642 3045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:05.654229 3047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:05.654484 3039 server_base.cc:1034] running on GCE node
I20260620 02:59:05.655093 3039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:05.656069 3039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:05.657296 3039 hybrid_clock.cc:648] HybridClock initialized: now 1781924345657239 us; error 48 us; skew 500 ppm
I20260620 02:59:05.657691 3039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:05 dist-test-slave-f1j0 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:05.660831 3039 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:59:05.662328 3039 webserver.cc:460] Webserver started at http://127.0.97.2:37487/ using document root <none> and password file <none>
I20260620 02:59:05.663020 3039 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:05.663178 3039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:05.663516 3039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:05.666625 3039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "90e248e52f7c4ad58a6a795ef6dde71f"
format_stamp: "Formatted at 2026-06-20 02:59:05 on dist-test-slave-f1j0"
server_key: "6949ce74cbd10219dc393c05eeb7e5dc"
server_key_iv: "f0fe2eadc65ebebcc2a4822a170c7f74"
server_key_version: "encryptionkey@0"
I20260620 02:59:05.667420 3039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "90e248e52f7c4ad58a6a795ef6dde71f"
format_stamp: "Formatted at 2026-06-20 02:59:05 on dist-test-slave-f1j0"
server_key: "6949ce74cbd10219dc393c05eeb7e5dc"
server_key_iv: "f0fe2eadc65ebebcc2a4822a170c7f74"
server_key_version: "encryptionkey@0"
I20260620 02:59:05.672616 3039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:59:05.676400 3054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:05.677942 3039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:59:05.678081 3039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "90e248e52f7c4ad58a6a795ef6dde71f"
format_stamp: "Formatted at 2026-06-20 02:59:05 on dist-test-slave-f1j0"
server_key: "6949ce74cbd10219dc393c05eeb7e5dc"
server_key_iv: "f0fe2eadc65ebebcc2a4822a170c7f74"
server_key_version: "encryptionkey@0"
I20260620 02:59:05.678262 3039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:05.691676 3039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:05.695580 3039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:05.695814 3039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:05.696652 3039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:05.698035 3039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:05.698102 3039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:05.698238 3039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:05.698303 3039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:05.730911 3039 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42481
I20260620 02:59:05.730947 3166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42481 every 8 connection(s)
I20260620 02:59:05.732242 3039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:59:05.737838 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 3039
I20260620 02:59:05.738000 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:59:05.738514 388 external_mini_cluster.cc:1428] Setting key 4363e45ee1fb2833f613162fc49dcff6
Jun 20 02:59:05 dist-test-slave-f1j0 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:05.755297 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:05.735023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34931 (local address 127.0.97.62:39637)
0620 02:59:05.735210 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:05.735214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:05.736418 (+ 1204us) server_negotiation.cc:407] Connection header received
0620 02:59:05.738636 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:05.738644 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:05.738744 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:05.738832 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:05.740801 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:05.741516 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:05.744032 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:05.744429 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:05.744525 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:05.748275 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:05.748284 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:05.748292 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:05.748338 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:05.751962 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:05.752657 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:05.752665 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:05.752671 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:05.752827 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:05.753229 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:05.753237 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:05.753243 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:05.753655 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:05.753809 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:05.754588 (+ 779us) server_negotiation.cc:299] Negotiation successful
0620 02:59:05.754843 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:59:05.756040 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:05.735538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39637 (local address 127.0.97.2:34931)
0620 02:59:05.736241 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:05.736289 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:05.738313 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:05.739110 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:05.739123 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:05.739909 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:05.740603 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:05.740624 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:05.741709 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:05.741718 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:05.743876 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:05.743893 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:05.744649 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:05.744658 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:05.744972 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:05.745796 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:05.745829 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:05.748035 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:05.752110 (+ 4075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:05.752117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:05.752131 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:05.752533 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:05.752937 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:05.752940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:05.752942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:05.753112 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:05.753776 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:05.753782 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:05.754144 (+ 362us) client_negotiation.cc:769] Sending connection context
0620 02:59:05.754437 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 02:59:05.754839 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":283,"threads_started":1}
I20260620 02:59:05.757761 3167 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39637
I20260620 02:59:05.758062 3167 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:59:05.758826 3167 heartbeater.cc:511] Master 127.0.97.62:39637 requested a full tablet report, sending...
I20260620 02:59:05.760352 2844 ts_manager.cc:194] Registered new tserver with Master: 90e248e52f7c4ad58a6a795ef6dde71f (127.0.97.2:42481)
I20260620 02:59:05.761216 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:59:05.785127 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39637
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:06.103253 3172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:06.103605 3172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:06.103777 3172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:06.111116 3172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:06.111286 3172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:06.111477 3172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:59:06.124562 3172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:06.126019 3172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:06.127332 3172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:06.146164 3178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:06.146158 3177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:06.146149 3180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:06.147073 3172 server_base.cc:1034] running on GCE node
I20260620 02:59:06.147722 3172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:06.148708 3172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:06.149936 3172 hybrid_clock.cc:648] HybridClock initialized: now 1781924346149888 us; error 44 us; skew 500 ppm
I20260620 02:59:06.150261 3172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:06 dist-test-slave-f1j0 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:06.153393 3172 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:59:06.154806 3172 webserver.cc:460] Webserver started at http://127.0.97.3:39643/ using document root <none> and password file <none>
I20260620 02:59:06.155489 3172 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:06.155637 3172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:06.155944 3172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:06.158833 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "032efbc9de4b47308640b0f760b29014"
format_stamp: "Formatted at 2026-06-20 02:59:06 on dist-test-slave-f1j0"
server_key: "06533150c7a86581c18373a907fdecd9"
server_key_iv: "6538f94811ce72061decbba4d784fd62"
server_key_version: "encryptionkey@0"
I20260620 02:59:06.159623 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "032efbc9de4b47308640b0f760b29014"
format_stamp: "Formatted at 2026-06-20 02:59:06 on dist-test-slave-f1j0"
server_key: "06533150c7a86581c18373a907fdecd9"
server_key_iv: "6538f94811ce72061decbba4d784fd62"
server_key_version: "encryptionkey@0"
I20260620 02:59:06.164535 3172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 02:59:06.168119 3187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:06.169368 3172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 02:59:06.169536 3172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "032efbc9de4b47308640b0f760b29014"
format_stamp: "Formatted at 2026-06-20 02:59:06 on dist-test-slave-f1j0"
server_key: "06533150c7a86581c18373a907fdecd9"
server_key_iv: "6538f94811ce72061decbba4d784fd62"
server_key_version: "encryptionkey@0"
I20260620 02:59:06.169806 3172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:06.186101 3172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:06.190166 3172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:06.190464 3172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:06.191411 3172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:06.192775 3172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:06.192873 3172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:06.193014 3172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:06.193086 3172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:06.225322 3172 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45395
I20260620 02:59:06.225364 3299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45395 every 8 connection(s)
I20260620 02:59:06.226629 3172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:59:06.232575 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 3172
I20260620 02:59:06.232751 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:59:06.233273 388 external_mini_cluster.cc:1428] Setting key 2c791b7aed824fabeba959832dd7c6f3
Jun 20 02:59:06 dist-test-slave-f1j0 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:06.246402 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:06.229165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54341 (local address 127.0.97.62:39637)
0620 02:59:06.229335 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:06.229340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:06.230510 (+ 1170us) server_negotiation.cc:407] Connection header received
0620 02:59:06.232357 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:06.232361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:06.232423 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:06.232505 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:06.234525 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:06.235540 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:06.237275 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:06.237707 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:06.237801 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:06.241091 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:06.241096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:06.241099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:06.241131 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:06.243620 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:06.244316 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:06.244319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:06.244322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:06.244439 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:06.244814 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:06.244817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:06.244819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:06.245059 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:06.245161 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:06.245978 (+ 817us) server_negotiation.cc:299] Negotiation successful
0620 02:59:06.246140 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:59:06.247532 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:06.229671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39637 (local address 127.0.97.3:54341)
0620 02:59:06.230355 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:06.230396 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:06.232015 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:06.232798 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:06.232811 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:06.233562 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:06.234354 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:06.234380 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:06.235693 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:06.235697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:06.237134 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:06.237145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:06.237887 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:06.237891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:06.238085 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:06.238719 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:06.238740 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:06.240884 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:06.243757 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:06.243764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:06.243779 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:06.244203 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:06.244533 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:06.244537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:06.244539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:06.244717 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:06.245161 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:06.245167 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:06.245539 (+ 372us) client_negotiation.cc:769] Sending connection context
0620 02:59:06.245863 (+ 324us) client_negotiation.cc:240] Negotiation successful
0620 02:59:06.246262 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":264,"threads_started":1}
I20260620 02:59:06.249382 3300 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39637
I20260620 02:59:06.249734 3300 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:06.250628 3300 heartbeater.cc:511] Master 127.0.97.62:39637 requested a full tablet report, sending...
I20260620 02:59:06.252269 2843 ts_manager.cc:194] Registered new tserver with Master: 032efbc9de4b47308640b0f760b29014 (127.0.97.3:45395)
I20260620 02:59:06.253456 2843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54341
I20260620 02:59:06.261279 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:59:06.282114 3033 heartbeater.cc:503] Master 127.0.97.62:39637 was elected leader, sending a full tablet report...
W20260620 02:59:06.291711 3035 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:06.289092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44478 (local address 127.0.97.62:39637)
0620 02:59:06.289266 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:06.289271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:06.289802 (+ 531us) server_negotiation.cc:407] Connection header received
0620 02:59:06.290018 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:06.290023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:06.290083 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:06.290186 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:06.291426 (+ 1240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44478: BlockingRecv error: recv got EOF from 127.0.0.1:44478 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:59:06.294853 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 2905
W20260620 02:59:06.297631 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 2905: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:35553: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:59:06.297823 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 2905
I20260620 02:59:06.313053 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 3039
W20260620 02:59:06.315613 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 3039: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:42481: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:59:06.315721 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 3039
I20260620 02:59:06.331007 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 3172
W20260620 02:59:06.333844 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 3172: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:45395: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:59:06.334017 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 3172
I20260620 02:59:06.349223 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 2812
I20260620 02:59:06.350538 2874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:06.453438 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 2812
2026-06-20T02:59:06Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3504 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:59:06 dist-test-slave-f1j0 krb5kdc[3317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:59:06 dist-test-slave-f1j0 krb5kdc[3317](info): set up 2 sockets
Jun 20 02:59:06 dist-test-slave-f1j0 krb5kdc[3317](info): commencing operation
krb5kdc: starting...
W20260620 02:59:07.537937 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 02:59:07 dist-test-slave-f1j0 krb5kdc[3317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:59:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:59:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:59:07.733304 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44151
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:08.045466 3330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:08.045830 3330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:08.046032 3330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:08.053277 3330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:59:08.053375 3330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:08.053517 3330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:08.053640 3330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:59:08.053730 3330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:59:08.063736 3330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:08.065012 3330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:08.066356 3330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:08.077971 3335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:08.078126 3338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:08.078096 3336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:08.079105 3330 server_base.cc:1034] running on GCE node
I20260620 02:59:08.079808 3330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:08.081089 3330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:08.082283 3330 hybrid_clock.cc:648] HybridClock initialized: now 1781924348082221 us; error 53 us; skew 500 ppm
I20260620 02:59:08.082752 3330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:08 dist-test-slave-f1j0 krb5kdc[3317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:08.085852 3330 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:59:08.087272 3330 webserver.cc:460] Webserver started at http://127.0.97.62:42687/ using document root <none> and password file <none>
I20260620 02:59:08.087967 3330 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:08.088161 3330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:08.088477 3330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:08.091586 3330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "b1a1c2b7105441a9b10bdf1f15db8206"
format_stamp: "Formatted at 2026-06-20 02:59:08 on dist-test-slave-f1j0"
server_key: "8705ffae7dc739465a349572c4d4042f"
server_key_iv: "13f275791d75734bf337aae17516768d"
server_key_version: "encryptionkey@0"
I20260620 02:59:08.092381 3330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "b1a1c2b7105441a9b10bdf1f15db8206"
format_stamp: "Formatted at 2026-06-20 02:59:08 on dist-test-slave-f1j0"
server_key: "8705ffae7dc739465a349572c4d4042f"
server_key_iv: "13f275791d75734bf337aae17516768d"
server_key_version: "encryptionkey@0"
I20260620 02:59:08.097348 3330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:59:08.100716 3345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:08.102089 3330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:59:08.102276 3330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "b1a1c2b7105441a9b10bdf1f15db8206"
format_stamp: "Formatted at 2026-06-20 02:59:08 on dist-test-slave-f1j0"
server_key: "8705ffae7dc739465a349572c4d4042f"
server_key_iv: "13f275791d75734bf337aae17516768d"
server_key_version: "encryptionkey@0"
I20260620 02:59:08.102506 3330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:08.122184 3330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:08.126694 3330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:08.126981 3330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:08.147745 3330 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44151
I20260620 02:59:08.147768 3396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44151 every 8 connection(s)
I20260620 02:59:08.149152 3330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:59:08.151185 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 3330
I20260620 02:59:08.151412 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 02:59:08.152221 388 external_mini_cluster.cc:1428] Setting key ad2fd58457ed136c701ebf58eefe2e05
I20260620 02:59:08.154129 3397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:08.167097 3397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206: Bootstrap starting.
Jun 20 02:59:08 dist-test-slave-f1j0 krb5kdc[3317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:08.171020 3397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:08.172336 3397 log.cc:824] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:08.175721 3397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206: No bootstrap required, opened a new log
I20260620 02:59:08.176826 3400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:08.154247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42360 (local address 127.0.97.62:44151)
0620 02:59:08.155085 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:08.155096 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:08.155142 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:59:08.156817 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:08.156839 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:08.157232 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:08.157719 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:08.159537 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:08.161564 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:08.163524 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:08.164656 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:08.164839 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:08.168766 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:08.168777 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:08.168799 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:08.168858 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:08.172676 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:08.173330 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:08.173339 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:08.173358 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:08.173618 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:08.174108 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:08.174112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:08.174114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:08.174639 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:08.174842 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:08.175335 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 02:59:08.175727 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":384,"threads_started":1}
I20260620 02:59:08.180868 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:59:08.185247 3397 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a1c2b7105441a9b10bdf1f15db8206" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44151 } }
I20260620 02:59:08.185763 3397 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:08.185859 3397 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1a1c2b7105441a9b10bdf1f15db8206, State: Initialized, Role: FOLLOWER
I20260620 02:59:08.186518 3397 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a1c2b7105441a9b10bdf1f15db8206" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44151 } }
I20260620 02:59:08.186834 3397 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:59:08.186980 3397 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:59:08.187151 3397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:08.189841 3397 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a1c2b7105441a9b10bdf1f15db8206" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44151 } }
I20260620 02:59:08.190327 3397 leader_election.cc:302] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1a1c2b7105441a9b10bdf1f15db8206; no voters:
I20260620 02:59:08.190977 3397 leader_election.cc:288] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:59:08.191089 3403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:08.192198 3403 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206 [term 1 LEADER]: Becoming Leader. State: Replica: b1a1c2b7105441a9b10bdf1f15db8206, State: Running, Role: LEADER
I20260620 02:59:08.192770 3403 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a1c2b7105441a9b10bdf1f15db8206" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44151 } }
I20260620 02:59:08.193378 3397 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:59:08.195883 3404 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1a1c2b7105441a9b10bdf1f15db8206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a1c2b7105441a9b10bdf1f15db8206" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44151 } } }
I20260620 02:59:08.196291 3404 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:08.196844 3405 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1a1c2b7105441a9b10bdf1f15db8206. Latest consensus state: current_term: 1 leader_uuid: "b1a1c2b7105441a9b10bdf1f15db8206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a1c2b7105441a9b10bdf1f15db8206" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44151 } } }
I20260620 02:59:08.197234 3405 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:08.209383 3408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:59:08.216001 3408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:59:08.240377 3408 catalog_manager.cc:1347] Generated new cluster ID: 9330c0a465dc401ea4c22a9abb66d72d
I20260620 02:59:08.240542 3408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:59:08.252177 3400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:08.239590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.0.97.62:44151)
0620 02:59:08.239784 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:08.239790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:08.243301 (+ 3511us) server_negotiation.cc:407] Connection header received
0620 02:59:08.243484 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:08.243490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:08.243577 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:08.243691 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:08.244930 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:08.246093 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:08.247985 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:08.248657 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:08.248794 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:08.250185 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:08.250193 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:08.250198 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:59:08.251300 (+ 1102us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:08.251378 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:59:08.251413 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:08.251522 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:08.251706 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 02:59:08.251798 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:59:08.258172 3408 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:59:08.259756 3408 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:59:08.272603 3408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1a1c2b7105441a9b10bdf1f15db8206: Generated new TSK 0
I20260620 02:59:08.273687 3408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:59:08.282367 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 3330
I20260620 02:59:08.283416 3392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:08.375599 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 3330
2026-06-20T02:59:08Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1916 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:59:08 dist-test-slave-f1j0 krb5kdc[3431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:59:08 dist-test-slave-f1j0 krb5kdc[3431](info): set up 2 sockets
Jun 20 02:59:08 dist-test-slave-f1j0 krb5kdc[3431](info): commencing operation
krb5kdc: starting...
W20260620 02:59:09.458321 388 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 02:59:09 dist-test-slave-f1j0 krb5kdc[3431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:59:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:59:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:59:09.650238 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34151
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:09.974200 3444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:09.974483 3444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:09.974648 3444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:09.981990 3444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:59:09.982110 3444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:09.982201 3444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:09.982309 3444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:59:09.982394 3444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:59:09.992503 3444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:09.994006 3444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:09.995313 3444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:10.007701 3450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:10.007791 3449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:10.007827 3452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:10.008616 3444 server_base.cc:1034] running on GCE node
I20260620 02:59:10.009480 3444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:10.011219 3444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:10.012437 3444 hybrid_clock.cc:648] HybridClock initialized: now 1781924350012385 us; error 48 us; skew 500 ppm
I20260620 02:59:10.012801 3444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:10 dist-test-slave-f1j0 krb5kdc[3431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:10.016010 3444 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:59:10.017673 3444 webserver.cc:460] Webserver started at http://127.0.97.62:45203/ using document root <none> and password file <none>
I20260620 02:59:10.018396 3444 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:10.018498 3444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:10.018854 3444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:10.021963 3444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "12d3f2193489428da699bdc801e89c55"
format_stamp: "Formatted at 2026-06-20 02:59:10 on dist-test-slave-f1j0"
server_key: "6652a929626710e8fb7db8ca22dc9c4e"
server_key_iv: "1bd52c32decdd38fbe7abfb448349741"
server_key_version: "encryptionkey@0"
I20260620 02:59:10.022744 3444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "12d3f2193489428da699bdc801e89c55"
format_stamp: "Formatted at 2026-06-20 02:59:10 on dist-test-slave-f1j0"
server_key: "6652a929626710e8fb7db8ca22dc9c4e"
server_key_iv: "1bd52c32decdd38fbe7abfb448349741"
server_key_version: "encryptionkey@0"
I20260620 02:59:10.027949 3444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 02:59:10.031582 3459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:10.032950 3444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:59:10.033138 3444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "12d3f2193489428da699bdc801e89c55"
format_stamp: "Formatted at 2026-06-20 02:59:10 on dist-test-slave-f1j0"
server_key: "6652a929626710e8fb7db8ca22dc9c4e"
server_key_iv: "1bd52c32decdd38fbe7abfb448349741"
server_key_version: "encryptionkey@0"
I20260620 02:59:10.033326 3444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:10.051337 3444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:10.055927 3444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:10.056267 3444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:10.074921 3444 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34151
I20260620 02:59:10.074954 3510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34151 every 8 connection(s)
I20260620 02:59:10.076325 3444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:59:10.080044 3511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:10.081784 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 3444
I20260620 02:59:10.082036 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 02:59:10.082813 388 external_mini_cluster.cc:1428] Setting key 4c788303484d3ac2d15792e008f6b664
I20260620 02:59:10.095009 3511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55: Bootstrap starting.
Jun 20 02:59:10 dist-test-slave-f1j0 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:10.098760 3511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:10.100034 3511 log.cc:824] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:10.103029 3511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55: No bootstrap required, opened a new log
I20260620 02:59:10.105775 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:10.084948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49098 (local address 127.0.97.62:34151)
0620 02:59:10.085791 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:10.085807 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:10.085849 (+ 42us) server_negotiation.cc:407] Connection header received
0620 02:59:10.087369 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:10.087390 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:10.087799 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:10.088169 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:10.089912 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:10.091205 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:10.092963 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:10.093723 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:10.093856 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:10.098429 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:10.098438 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:10.098454 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:10.098494 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:10.101673 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:10.102343 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:10.102360 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:10.102367 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:10.102524 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:10.102953 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:10.102960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:10.102963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:10.103543 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:10.103851 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:10.104266 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 02:59:10.104691 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":402,"threads_started":1}
I20260620 02:59:10.111047 3511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d3f2193489428da699bdc801e89c55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34151 } }
I20260620 02:59:10.111421 3511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:10.111552 3511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d3f2193489428da699bdc801e89c55, State: Initialized, Role: FOLLOWER
I20260620 02:59:10.112179 3511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d3f2193489428da699bdc801e89c55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34151 } }
I20260620 02:59:10.112452 3511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:59:10.112612 3511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:59:10.112797 3511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:10.116586 3511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d3f2193489428da699bdc801e89c55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34151 } }
I20260620 02:59:10.117249 3511 leader_election.cc:302] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12d3f2193489428da699bdc801e89c55; no voters:
I20260620 02:59:10.118065 3511 leader_election.cc:288] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:59:10.118175 3516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:10.120187 3511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:59:10.121490 3516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55 [term 1 LEADER]: Becoming Leader. State: Replica: 12d3f2193489428da699bdc801e89c55, State: Running, Role: LEADER
I20260620 02:59:10.122153 3516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d3f2193489428da699bdc801e89c55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34151 } }
I20260620 02:59:10.127805 3517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12d3f2193489428da699bdc801e89c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d3f2193489428da699bdc801e89c55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34151 } } }
I20260620 02:59:10.128234 3517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:10.129562 3518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12d3f2193489428da699bdc801e89c55. Latest consensus state: current_term: 1 leader_uuid: "12d3f2193489428da699bdc801e89c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d3f2193489428da699bdc801e89c55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34151 } } }
I20260620 02:59:10.129940 3518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:10.132210 3525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:59:10.139282 3525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:59:10.150671 3525 catalog_manager.cc:1347] Generated new cluster ID: 46b2473020e948c68ef9367b141b3807
I20260620 02:59:10.150797 3525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:59:10.179324 3525 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:59:10.181191 3525 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:59:10.189805 3525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12d3f2193489428da699bdc801e89c55: Generated new TSK 0
I20260620 02:59:10.190692 3525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:59:10.245095 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34151
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:10.571841 3538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:10.572126 3538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:10.572220 3538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:10.579309 3538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:10.579420 3538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:10.579532 3538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:59:10.589103 3538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:10.590412 3538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:10.591709 3538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:10.606473 3546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:10.606500 3544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:10.606878 3543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:10.608429 3538 server_base.cc:1034] running on GCE node
I20260620 02:59:10.609102 3538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:10.610154 3538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:10.611392 3538 hybrid_clock.cc:648] HybridClock initialized: now 1781924350611342 us; error 44 us; skew 500 ppm
I20260620 02:59:10.611737 3538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:10 dist-test-slave-f1j0 krb5kdc[3431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:10.615185 3538 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:59:10.616695 3538 webserver.cc:460] Webserver started at http://127.0.97.1:37233/ using document root <none> and password file <none>
I20260620 02:59:10.617425 3538 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:10.617615 3538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:10.617934 3538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:10.621155 3538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "845d3bde27da44759e209f7340dd91a2"
format_stamp: "Formatted at 2026-06-20 02:59:10 on dist-test-slave-f1j0"
server_key: "34b9dbb46558e3d3dde748ba88525c30"
server_key_iv: "e73a4d7eeaf6f3433ac700f31947b07a"
server_key_version: "encryptionkey@0"
I20260620 02:59:10.622051 3538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "845d3bde27da44759e209f7340dd91a2"
format_stamp: "Formatted at 2026-06-20 02:59:10 on dist-test-slave-f1j0"
server_key: "34b9dbb46558e3d3dde748ba88525c30"
server_key_iv: "e73a4d7eeaf6f3433ac700f31947b07a"
server_key_version: "encryptionkey@0"
I20260620 02:59:10.627069 3538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 02:59:10.630594 3553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:10.631903 3538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:59:10.632066 3538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "845d3bde27da44759e209f7340dd91a2"
format_stamp: "Formatted at 2026-06-20 02:59:10 on dist-test-slave-f1j0"
server_key: "34b9dbb46558e3d3dde748ba88525c30"
server_key_iv: "e73a4d7eeaf6f3433ac700f31947b07a"
server_key_version: "encryptionkey@0"
I20260620 02:59:10.632254 3538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:10.666534 3538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:10.670989 3538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:10.671304 3538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:10.672257 3538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:10.673599 3538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:10.673669 3538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:10.673801 3538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:10.673877 3538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:10.706768 3538 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42729
I20260620 02:59:10.706858 3665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42729 every 8 connection(s)
I20260620 02:59:10.708086 3538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:59:10.716962 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 3538
I20260620 02:59:10.717231 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:59:10.717897 388 external_mini_cluster.cc:1428] Setting key 1e93f19e4f72c9f9f7cd6290a278761a
Jun 20 02:59:10 dist-test-slave-f1j0 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:10.727789 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:10.710736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56431 (local address 127.0.97.62:34151)
0620 02:59:10.711194 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:10.711199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:10.712107 (+ 908us) server_negotiation.cc:407] Connection header received
0620 02:59:10.713929 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:10.713934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:10.714028 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:10.714160 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:10.715606 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:10.716356 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:10.718202 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:10.718598 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:10.718743 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:10.722212 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:10.722217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:10.722220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:10.722251 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:10.724723 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:10.725440 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:10.725451 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:10.725454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:10.725562 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:10.726072 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:10.726077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:10.726080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:10.726356 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:10.726509 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:10.727213 (+ 704us) server_negotiation.cc:299] Negotiation successful
0620 02:59:10.727416 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":247,"threads_started":1}
I20260620 02:59:10.728791 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:10.711268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34151 (local address 127.0.97.1:56431)
0620 02:59:10.711949 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:10.711993 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:10.713671 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:10.714384 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:10.714392 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:10.714892 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:10.715436 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:10.715452 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:10.716571 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:10.716574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:10.718026 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:10.718041 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:10.718822 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:10.718831 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:10.719077 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:10.719761 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:10.719783 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:10.721986 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:10.724863 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:10.724872 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:10.724889 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:10.725310 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:10.725764 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:10.725770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:10.725774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:10.725954 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:10.726491 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:10.726498 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:10.726831 (+ 333us) client_negotiation.cc:769] Sending connection context
0620 02:59:10.727104 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 02:59:10.727610 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":271,"threads_started":1}
I20260620 02:59:10.730434 3666 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34151
I20260620 02:59:10.730762 3666 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:10.731670 3666 heartbeater.cc:511] Master 127.0.97.62:34151 requested a full tablet report, sending...
I20260620 02:59:10.733950 3476 ts_manager.cc:194] Registered new tserver with Master: 845d3bde27da44759e209f7340dd91a2 (127.0.97.1:42729)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:59:10.735534 3476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:59:10.760317 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34151
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:11.089824 3672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:11.090178 3672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:11.090341 3672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:11.097949 3672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:11.098099 3672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:11.098273 3672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:59:11.108398 3672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:11.109783 3672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:11.111101 3672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:11.124337 3677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:11.124423 3678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:11.124373 3680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:11.125669 3672 server_base.cc:1034] running on GCE node
I20260620 02:59:11.126302 3672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:11.127270 3672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:11.128515 3672 hybrid_clock.cc:648] HybridClock initialized: now 1781924351128470 us; error 57 us; skew 500 ppm
I20260620 02:59:11.128840 3672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:11 dist-test-slave-f1j0 krb5kdc[3431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:11.132072 3672 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:59:11.133543 3672 webserver.cc:460] Webserver started at http://127.0.97.2:35533/ using document root <none> and password file <none>
I20260620 02:59:11.134413 3672 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:11.134550 3672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:11.134812 3672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:11.138041 3672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "858c840d47f64cd3b3d7e909f29d0cfc"
format_stamp: "Formatted at 2026-06-20 02:59:11 on dist-test-slave-f1j0"
server_key: "7ea8480ba46e5fbb3adc22ce2153a118"
server_key_iv: "59cc5a582345a28ca7776be0946fefc3"
server_key_version: "encryptionkey@0"
I20260620 02:59:11.138752 3672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "858c840d47f64cd3b3d7e909f29d0cfc"
format_stamp: "Formatted at 2026-06-20 02:59:11 on dist-test-slave-f1j0"
server_key: "7ea8480ba46e5fbb3adc22ce2153a118"
server_key_iv: "59cc5a582345a28ca7776be0946fefc3"
server_key_version: "encryptionkey@0"
I20260620 02:59:11.143616 3672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260620 02:59:11.146854 3687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:11.148177 3672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:59:11.148304 3672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "858c840d47f64cd3b3d7e909f29d0cfc"
format_stamp: "Formatted at 2026-06-20 02:59:11 on dist-test-slave-f1j0"
server_key: "7ea8480ba46e5fbb3adc22ce2153a118"
server_key_iv: "59cc5a582345a28ca7776be0946fefc3"
server_key_version: "encryptionkey@0"
I20260620 02:59:11.148451 3672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:11.171720 3672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:11.176153 3672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:11.176462 3672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:11.177414 3672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:11.178819 3672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:11.178891 3672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:11.179020 3672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:11.179071 3672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:11.212291 3672 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39959
I20260620 02:59:11.212340 3799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39959 every 8 connection(s)
I20260620 02:59:11.213549 3672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:59:11.214618 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 3672
I20260620 02:59:11.214823 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:59:11.215396 388 external_mini_cluster.cc:1428] Setting key 548262218e44759110f608e40b798b32
Jun 20 02:59:11 dist-test-slave-f1j0 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:59:11.237043 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:11.216623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49367 (local address 127.0.97.62:34151)
0620 02:59:11.216833 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:11.216840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:11.217998 (+ 1158us) server_negotiation.cc:407] Connection header received
0620 02:59:11.220902 (+ 2904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:11.220906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:11.221018 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:11.221128 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:11.223156 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:11.223825 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:11.226087 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:11.226492 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:11.226603 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:11.230922 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:11.230927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:11.230930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:11.230961 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:11.233716 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:11.234449 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:11.234454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:11.234458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:11.234588 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:11.235091 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:11.235095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:11.235099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:11.235369 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:11.235564 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:11.236404 (+ 840us) server_negotiation.cc:299] Negotiation successful
0620 02:59:11.236663 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 02:59:11.238261 3802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:11.216983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34151 (local address 127.0.97.2:49367)
0620 02:59:11.217835 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:11.217881 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:11.220596 (+ 2715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:11.221369 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:11.221382 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:11.222197 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:11.222978 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:11.223005 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:11.223973 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:11.223984 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:11.225924 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:11.225941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:11.226730 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:11.226739 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:11.227059 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:11.227789 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:11.227821 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:11.230671 (+ 2850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:11.233861 (+ 3190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:11.233869 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:11.233887 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:11.234318 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:11.234716 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:11.234721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:11.234725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:11.234967 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:11.235532 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:11.235540 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:11.235996 (+ 456us) client_negotiation.cc:769] Sending connection context
0620 02:59:11.236279 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 02:59:11.236862 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":314,"threads_started":1}
I20260620 02:59:11.239686 3800 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34151
I20260620 02:59:11.239992 3800 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:11.240701 3800 heartbeater.cc:511] Master 127.0.97.62:34151 requested a full tablet report, sending...
I20260620 02:59:11.242202 3476 ts_manager.cc:194] Registered new tserver with Master: 858c840d47f64cd3b3d7e909f29d0cfc (127.0.97.2:39959)
I20260620 02:59:11.242990 3476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:59:11.260816 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34151
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:11.576951 3805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:11.577232 3805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:11.577401 3805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:11.584730 3805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:11.584846 3805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:11.585048 3805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:59:11.594913 3805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:11.596256 3805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:11.597640 3805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:11.611011 3810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:11.611059 3811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:11.611510 3813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:11.612283 3805 server_base.cc:1034] running on GCE node
I20260620 02:59:11.612895 3805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:11.613942 3805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:11.615170 3805 hybrid_clock.cc:648] HybridClock initialized: now 1781924351615137 us; error 47 us; skew 500 ppm
I20260620 02:59:11.615489 3805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:11 dist-test-slave-f1j0 krb5kdc[3431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:11.618763 3805 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:59:11.620222 3805 webserver.cc:460] Webserver started at http://127.0.97.3:44967/ using document root <none> and password file <none>
I20260620 02:59:11.620914 3805 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:11.621064 3805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:11.621371 3805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:11.624248 3805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "22c4a8d999cf4054aad392a0530845ec"
format_stamp: "Formatted at 2026-06-20 02:59:11 on dist-test-slave-f1j0"
server_key: "1c6e541a5f9fabbbae2fc5e73c6abc5f"
server_key_iv: "f8c4c34fb7120da607bee3c3bd7afb61"
server_key_version: "encryptionkey@0"
I20260620 02:59:11.625025 3805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "22c4a8d999cf4054aad392a0530845ec"
format_stamp: "Formatted at 2026-06-20 02:59:11 on dist-test-slave-f1j0"
server_key: "1c6e541a5f9fabbbae2fc5e73c6abc5f"
server_key_iv: "f8c4c34fb7120da607bee3c3bd7afb61"
server_key_version: "encryptionkey@0"
I20260620 02:59:11.630445 3805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 02:59:11.633806 3820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:11.635170 3805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:59:11.635358 3805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "22c4a8d999cf4054aad392a0530845ec"
format_stamp: "Formatted at 2026-06-20 02:59:11 on dist-test-slave-f1j0"
server_key: "1c6e541a5f9fabbbae2fc5e73c6abc5f"
server_key_iv: "f8c4c34fb7120da607bee3c3bd7afb61"
server_key_version: "encryptionkey@0"
I20260620 02:59:11.635603 3805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:11.647322 3805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:11.651692 3805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:11.652019 3805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:11.653039 3805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:11.654373 3805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:11.654479 3805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:11.654630 3805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:11.654706 3805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:11.687002 3805 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44341
I20260620 02:59:11.687044 3932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44341 every 8 connection(s)
I20260620 02:59:11.688329 3805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:59:11.688673 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 3805
I20260620 02:59:11.688881 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:59:11.689504 388 external_mini_cluster.cc:1428] Setting key 36447e3075b581918405efcd16409675
Jun 20 02:59:11 dist-test-slave-f1j0 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:11.707835 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:11.691016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48165 (local address 127.0.97.62:34151)
0620 02:59:11.691237 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:11.691244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:11.692672 (+ 1428us) server_negotiation.cc:407] Connection header received
0620 02:59:11.694578 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:11.694581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:11.694647 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:11.694729 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:11.696134 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:11.696878 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:11.698420 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:11.698791 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:11.698884 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:11.702183 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:11.702188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:11.702191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:11.702221 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:11.704795 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:11.705428 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:11.705432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:11.705434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:11.705541 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:11.706020 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:11.706024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:11.706027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:11.706416 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:11.706555 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:11.707237 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 02:59:11.707489 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:59:11.708484 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:11.691752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34151 (local address 127.0.97.3:48165)
0620 02:59:11.692543 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:11.692588 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:11.694345 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:11.694941 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:11.694950 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:11.695432 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:11.695992 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:11.696008 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:11.697019 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:11.697023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:11.698300 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:11.698315 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:11.698970 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:11.698974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:11.699163 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:11.699796 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:11.699817 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:11.701993 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:11.704917 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:11.704923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:11.704935 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:11.705324 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:11.705696 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:11.705703 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:11.705710 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:11.705909 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:11.706532 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:11.706538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:11.706841 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 02:59:11.707051 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 02:59:11.707410 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":346,"threads_started":1}
I20260620 02:59:11.709955 3933 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34151
I20260620 02:59:11.710276 3933 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:11.711020 3933 heartbeater.cc:511] Master 127.0.97.62:34151 requested a full tablet report, sending...
I20260620 02:59:11.712376 3476 ts_manager.cc:194] Registered new tserver with Master: 22c4a8d999cf4054aad392a0530845ec (127.0.97.3:44341)
I20260620 02:59:11.713158 3476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48165
I20260620 02:59:11.716930 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:59:11.738905 3666 heartbeater.cc:503] Master 127.0.97.62:34151 was elected leader, sending a full tablet report...
Jun 20 02:59:11 dist-test-slave-f1j0 krb5kdc[3431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924351, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 02:59:11 dist-test-slave-f1j0 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924351, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:11.762988 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:11.750111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49110 (local address 127.0.97.62:34151)
0620 02:59:11.750349 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:11.750356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:11.750668 (+ 312us) server_negotiation.cc:407] Connection header received
0620 02:59:11.750812 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:11.750816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:11.750877 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:11.750961 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:11.752355 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:11.753122 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:11.754381 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:11.754758 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:11.754845 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:11.758342 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:11.758347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:11.758350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:11.758380 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:11.760832 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:11.761344 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:11.761347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:11.761349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:11.761458 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:11.761814 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:11.761817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:11.761820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:11.762102 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:11.762274 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:11.762505 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 02:59:11.762662 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 02:59:11.763512 3476 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49110
I20260620 02:59:11.765005 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 3538
Jun 20 02:59:11 dist-test-slave-f1j0 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924351, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:59:11.780293 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:11.766039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36154 (local address 127.0.97.1:42729)
0620 02:59:11.766685 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:11.766692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:11.766718 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:59:11.766778 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:11.766785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:11.766983 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:11.767108 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:11.768139 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:11.769118 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:11.770293 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:11.770827 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:11.770920 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:11.774904 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:11.774913 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:11.774927 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:11.774956 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:11.777736 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:11.778401 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:11.778411 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:11.778419 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:11.778531 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:11.778875 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:11.778880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:11.778884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:11.779190 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:11.779482 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:11.779716 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 02:59:11.779961 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":352,"threads_started":1}
W20260620 02:59:11.781526 3661 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36154
W20260620 02:59:11.782644 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 3538: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:59:11.782784 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 3538
I20260620 02:59:11.798193 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 3672
Jun 20 02:59:11 dist-test-slave-f1j0 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924351, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:59:11.812804 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:11.799049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.0.97.2:39959)
0620 02:59:11.799506 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:11.799514 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:11.799539 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:59:11.799595 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:11.799601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:11.799805 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:11.799932 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:11.801059 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:11.802114 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:11.803288 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:11.803796 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:11.803885 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:11.807846 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:11.807855 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:11.807869 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:11.807901 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:11.810489 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:11.811044 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:11.811050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:11.811052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:11.811175 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:11.811470 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:11.811473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:11.811475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:11.811775 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:11.811984 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:11.812249 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 02:59:11.812461 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":240,"threads_started":1}
W20260620 02:59:11.814042 3795 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35904
W20260620 02:59:11.815089 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 3672: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:59:11.815266 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 3672
I20260620 02:59:11.830442 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 3805
Jun 20 02:59:11 dist-test-slave-f1j0 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924351, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:59:11.846611 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:11.831302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33902 (local address 127.0.97.3:44341)
0620 02:59:11.831863 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:11.831870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:11.831898 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:59:11.831956 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:11.831962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:11.832164 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:11.832333 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:11.833625 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:11.834616 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:11.835912 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:11.836519 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:11.836671 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:11.841008 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:11.841025 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:11.841050 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:11.841134 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:11.844190 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:11.844793 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:11.844800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:11.844802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:11.844920 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:11.845219 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:11.845222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:11.845225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:11.845528 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:11.845890 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:11.846148 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 02:59:11.846335 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":258,"threads_started":1}
W20260620 02:59:11.847859 3928 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33902
W20260620 02:59:11.848979 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 3805: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:59:11.849134 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 3805
I20260620 02:59:11.864270 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 3444
I20260620 02:59:11.865440 3506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:11.965673 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 3444
2026-06-20T02:59:11Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3595 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:59:12 dist-test-slave-f1j0 krb5kdc[3950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:59:12 dist-test-slave-f1j0 krb5kdc[3950](info): set up 2 sockets
Jun 20 02:59:12 dist-test-slave-f1j0 krb5kdc[3950](info): commencing operation
krb5kdc: starting...
W20260620 02:59:13.046062 388 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 02:59:13 dist-test-slave-f1j0 krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:59:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:59:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:59:13.240944 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34247
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:13.555299 3963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:13.555660 3963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:13.555832 3963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:13.563143 3963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:59:13.563287 3963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:13.563393 3963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:13.563493 3963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:59:13.563575 3963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:59:13.573390 3963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:13.574777 3963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:13.576118 3963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:13.587931 3969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:13.588061 3968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:13.588059 3971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:13.588665 3963 server_base.cc:1034] running on GCE node
I20260620 02:59:13.589432 3963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:13.590860 3963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:13.592098 3963 hybrid_clock.cc:648] HybridClock initialized: now 1781924353592050 us; error 79 us; skew 500 ppm
I20260620 02:59:13.592458 3963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:13 dist-test-slave-f1j0 krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:13.595546 3963 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:59:13.596963 3963 webserver.cc:460] Webserver started at http://127.0.97.62:38153/ using document root <none> and password file <none>
I20260620 02:59:13.597769 3963 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:13.597916 3963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:13.598232 3963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:13.601110 3963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "33195a0e9643411d945999cd7468593e"
format_stamp: "Formatted at 2026-06-20 02:59:13 on dist-test-slave-f1j0"
server_key: "13cbadd61f4e4847dbcc16a3c5afa43a"
server_key_iv: "78569de5d9a2d33e625a8a66fa7c0a7e"
server_key_version: "encryptionkey@0"
I20260620 02:59:13.601975 3963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "33195a0e9643411d945999cd7468593e"
format_stamp: "Formatted at 2026-06-20 02:59:13 on dist-test-slave-f1j0"
server_key: "13cbadd61f4e4847dbcc16a3c5afa43a"
server_key_iv: "78569de5d9a2d33e625a8a66fa7c0a7e"
server_key_version: "encryptionkey@0"
I20260620 02:59:13.607090 3963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 02:59:13.610579 3978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:13.611901 3963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:59:13.612084 3963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "33195a0e9643411d945999cd7468593e"
format_stamp: "Formatted at 2026-06-20 02:59:13 on dist-test-slave-f1j0"
server_key: "13cbadd61f4e4847dbcc16a3c5afa43a"
server_key_iv: "78569de5d9a2d33e625a8a66fa7c0a7e"
server_key_version: "encryptionkey@0"
I20260620 02:59:13.612295 3963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:13.637794 3963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:13.641976 3963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:13.642273 3963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:13.660449 3963 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34247
I20260620 02:59:13.660485 4029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34247 every 8 connection(s)
I20260620 02:59:13.661855 3963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:59:13.665508 4030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:13.669077 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 3963
I20260620 02:59:13.669420 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 02:59:13.670082 388 external_mini_cluster.cc:1428] Setting key 39e187fc3564626df1e63c89ef858e10
I20260620 02:59:13.677982 4030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e: Bootstrap starting.
I20260620 02:59:13.681695 4030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:13.682763 4030 log.cc:824] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:13.685626 4030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e: No bootstrap required, opened a new log
Jun 20 02:59:13 dist-test-slave-f1j0 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:13.694589 4033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:13.672337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.0.97.62:34247)
0620 02:59:13.673140 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:13.673157 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:13.673209 (+ 52us) server_negotiation.cc:407] Connection header received
0620 02:59:13.676209 (+ 3000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:13.676239 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:13.676763 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:13.677193 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:13.678599 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:13.680658 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:13.681960 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:13.682672 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:13.682870 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:13.687231 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:13.687239 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:13.687253 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:13.687290 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:13.690515 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:13.691314 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:13.691321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:13.691327 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:13.691487 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:13.691887 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:13.691890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:13.691892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:13.692390 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:13.692596 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:13.693089 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 02:59:13.693484 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":344,"threads_started":1}
I20260620 02:59:13.694970 4030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33195a0e9643411d945999cd7468593e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34247 } }
I20260620 02:59:13.695344 4030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:13.695448 4030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33195a0e9643411d945999cd7468593e, State: Initialized, Role: FOLLOWER
I20260620 02:59:13.696043 4030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33195a0e9643411d945999cd7468593e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34247 } }
I20260620 02:59:13.696306 4030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:59:13.696455 4030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:59:13.696633 4030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:13.699051 4030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33195a0e9643411d945999cd7468593e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34247 } }
I20260620 02:59:13.699575 4030 leader_election.cc:302] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33195a0e9643411d945999cd7468593e; no voters:
I20260620 02:59:13.700182 4030 leader_election.cc:288] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:59:13.700330 4035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:13.701066 4035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e [term 1 LEADER]: Becoming Leader. State: Replica: 33195a0e9643411d945999cd7468593e, State: Running, Role: LEADER
I20260620 02:59:13.701558 4035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33195a0e9643411d945999cd7468593e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34247 } }
I20260620 02:59:13.702457 4030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:59:13.704502 4036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33195a0e9643411d945999cd7468593e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33195a0e9643411d945999cd7468593e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34247 } } }
I20260620 02:59:13.704945 4036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:13.705727 4037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33195a0e9643411d945999cd7468593e. Latest consensus state: current_term: 1 leader_uuid: "33195a0e9643411d945999cd7468593e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33195a0e9643411d945999cd7468593e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34247 } } }
I20260620 02:59:13.706207 4037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:13.706723 4042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:59:13.713532 4042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:59:13.722672 4042 catalog_manager.cc:1347] Generated new cluster ID: 88c2e33cd8144a4e827e9bcb466a9406
I20260620 02:59:13.722790 4042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:59:13.785063 4042 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:59:13.786924 4042 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:59:13.801362 4042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33195a0e9643411d945999cd7468593e: Generated new TSK 0
I20260620 02:59:13.802487 4042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:59:13.859520 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34247
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:14.181591 4056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:14.181880 4056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:14.182049 4056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:14.189281 4056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:14.189380 4056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:14.189611 4056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:59:14.199247 4056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:14.200523 4056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:14.201900 4056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:14.215423 4061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:14.215471 4062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:14.215665 4064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:14.216511 4056 server_base.cc:1034] running on GCE node
I20260620 02:59:14.217244 4056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:14.218243 4056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:14.219444 4056 hybrid_clock.cc:648] HybridClock initialized: now 1781924354219403 us; error 36 us; skew 500 ppm
I20260620 02:59:14.219750 4056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:14 dist-test-slave-f1j0 krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:14.222759 4056 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:59:14.224174 4056 webserver.cc:460] Webserver started at http://127.0.97.1:42401/ using document root <none> and password file <none>
I20260620 02:59:14.224835 4056 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:14.224978 4056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:14.225283 4056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:14.228291 4056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "e8bb6be93be247329eda153e17f677c6"
format_stamp: "Formatted at 2026-06-20 02:59:14 on dist-test-slave-f1j0"
server_key: "f02581d0680abbc9292bd25c086379a6"
server_key_iv: "ee714e20118b1b74f6966268daf8d776"
server_key_version: "encryptionkey@0"
I20260620 02:59:14.229059 4056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e8bb6be93be247329eda153e17f677c6"
format_stamp: "Formatted at 2026-06-20 02:59:14 on dist-test-slave-f1j0"
server_key: "f02581d0680abbc9292bd25c086379a6"
server_key_iv: "ee714e20118b1b74f6966268daf8d776"
server_key_version: "encryptionkey@0"
I20260620 02:59:14.233881 4056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20260620 02:59:14.237150 4071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:14.238598 4056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:59:14.238780 4056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "e8bb6be93be247329eda153e17f677c6"
format_stamp: "Formatted at 2026-06-20 02:59:14 on dist-test-slave-f1j0"
server_key: "f02581d0680abbc9292bd25c086379a6"
server_key_iv: "ee714e20118b1b74f6966268daf8d776"
server_key_version: "encryptionkey@0"
I20260620 02:59:14.238986 4056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:14.256300 4056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:14.260332 4056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:14.260643 4056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:14.261677 4056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:14.263006 4056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:14.263134 4056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:14.263296 4056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:14.263372 4056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:14.296094 4056 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45255
I20260620 02:59:14.296100 4183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45255 every 8 connection(s)
I20260620 02:59:14.297406 4056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:59:14.307775 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 4056
I20260620 02:59:14.307962 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:59:14.308818 388 external_mini_cluster.cc:1428] Setting key da0fabfa422091e30301f8762249538c
Jun 20 02:59:14 dist-test-slave-f1j0 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:14.318143 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:14.300107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57865 (local address 127.0.97.62:34247)
0620 02:59:14.300663 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:14.300668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:14.301656 (+ 988us) server_negotiation.cc:407] Connection header received
0620 02:59:14.303477 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:14.303483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:14.303565 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:14.303685 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:14.305222 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:14.305987 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:14.307541 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:14.308072 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:14.308187 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:14.312294 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:14.312298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:14.312302 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:14.312330 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:14.314897 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:14.315789 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:14.315793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:14.315796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:14.315903 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:14.316407 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:14.316411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:14.316413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:14.316727 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:14.316833 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:14.317758 (+ 925us) server_negotiation.cc:299] Negotiation successful
0620 02:59:14.317908 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":277,"threads_started":1}
I20260620 02:59:14.319680 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:14.300753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34247 (local address 127.0.97.1:57865)
0620 02:59:14.301474 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:14.301511 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:14.303195 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:14.303958 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:14.303970 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:14.304518 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:14.305049 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:14.305065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:14.306146 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:14.306150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:14.307371 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:14.307384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:14.308317 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:14.308324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:14.308617 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:14.309400 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:14.309435 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:14.312089 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:14.315032 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:14.315043 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:14.315064 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:14.315677 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:14.316010 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:14.316017 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:14.316024 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:14.316293 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:14.316856 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:14.316865 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:14.317316 (+ 451us) client_negotiation.cc:769] Sending connection context
0620 02:59:14.317690 (+ 374us) client_negotiation.cc:240] Negotiation successful
0620 02:59:14.318257 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":287,"threads_started":1}
I20260620 02:59:14.321238 4184 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34247
I20260620 02:59:14.321540 4184 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:14.322439 4184 heartbeater.cc:511] Master 127.0.97.62:34247 requested a full tablet report, sending...
I20260620 02:59:14.324759 3995 ts_manager.cc:194] Registered new tserver with Master: e8bb6be93be247329eda153e17f677c6 (127.0.97.1:45255)
I20260620 02:59:14.326969 3995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57865
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:59:14.357081 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34247
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:14.689234 4190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:14.689563 4190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:14.689762 4190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:14.696887 4190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:14.697031 4190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:14.697213 4190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:59:14.706727 4190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:14.708034 4190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:14.709332 4190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:14.722566 4195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:14.722608 4198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:14.722524 4196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:14.723608 4190 server_base.cc:1034] running on GCE node
I20260620 02:59:14.724220 4190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:14.725212 4190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:14.726408 4190 hybrid_clock.cc:648] HybridClock initialized: now 1781924354726362 us; error 34 us; skew 500 ppm
I20260620 02:59:14.726732 4190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:14 dist-test-slave-f1j0 krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:14.732887 4190 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:59:14.734462 4190 webserver.cc:460] Webserver started at http://127.0.97.2:35627/ using document root <none> and password file <none>
I20260620 02:59:14.735105 4190 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:14.735210 4190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:14.735451 4190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:14.738440 4190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "359653970fde48208e10754a65c44608"
format_stamp: "Formatted at 2026-06-20 02:59:14 on dist-test-slave-f1j0"
server_key: "1d5ba889ffb07cc31cdc9414e0dd4578"
server_key_iv: "7758762a2b6039266f576b8c51eccce4"
server_key_version: "encryptionkey@0"
I20260620 02:59:14.739148 4190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "359653970fde48208e10754a65c44608"
format_stamp: "Formatted at 2026-06-20 02:59:14 on dist-test-slave-f1j0"
server_key: "1d5ba889ffb07cc31cdc9414e0dd4578"
server_key_iv: "7758762a2b6039266f576b8c51eccce4"
server_key_version: "encryptionkey@0"
I20260620 02:59:14.744246 4190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:59:14.747627 4205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:14.748911 4190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:59:14.749086 4190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "359653970fde48208e10754a65c44608"
format_stamp: "Formatted at 2026-06-20 02:59:14 on dist-test-slave-f1j0"
server_key: "1d5ba889ffb07cc31cdc9414e0dd4578"
server_key_iv: "7758762a2b6039266f576b8c51eccce4"
server_key_version: "encryptionkey@0"
I20260620 02:59:14.749286 4190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:14.765429 4190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:14.769821 4190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:14.770104 4190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:14.771035 4190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:14.772403 4190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:14.772504 4190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:14.772648 4190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:14.772723 4190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:14.805928 4190 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36025
I20260620 02:59:14.805974 4317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36025 every 8 connection(s)
I20260620 02:59:14.807258 4190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:59:14.815361 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 4190
I20260620 02:59:14.815572 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:59:14.816072 388 external_mini_cluster.cc:1428] Setting key 377182a3d59a56e936f6be3ecaf76f52
Jun 20 02:59:14 dist-test-slave-f1j0 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:14.829100 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:14.809896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60915 (local address 127.0.97.62:34247)
0620 02:59:14.810067 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:14.810072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:14.811459 (+ 1387us) server_negotiation.cc:407] Connection header received
0620 02:59:14.813236 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:14.813241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:14.813328 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:14.813454 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:14.814967 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:14.815803 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:14.817712 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:14.818268 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:14.818369 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:14.822835 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:14.822840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:14.822843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:14.822873 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:14.825926 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:14.826619 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:14.826628 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:14.826634 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:14.826786 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:14.827182 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:14.827190 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:14.827197 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:14.827566 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:14.827720 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:14.828355 (+ 635us) server_negotiation.cc:299] Negotiation successful
0620 02:59:14.828610 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:59:14.829790 4320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:14.810475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34247 (local address 127.0.97.2:60915)
0620 02:59:14.811297 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:14.811338 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:14.812966 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:14.813728 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:14.813737 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:14.814250 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:14.814802 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:14.814818 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:14.815954 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:14.815978 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:14.817499 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:14.817510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:14.818489 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:14.818497 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:14.818842 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:14.819565 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:14.819600 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:14.822604 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:14.826059 (+ 3455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:14.826066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:14.826079 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:14.826491 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:14.826889 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:14.826892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:14.826895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:14.827061 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:14.827680 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:14.827685 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:14.827991 (+ 306us) client_negotiation.cc:769] Sending connection context
0620 02:59:14.828229 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 02:59:14.828591 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":304,"threads_started":1}
I20260620 02:59:14.831327 4318 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34247
I20260620 02:59:14.831663 4318 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:14.832597 4318 heartbeater.cc:511] Master 127.0.97.62:34247 requested a full tablet report, sending...
I20260620 02:59:14.834339 3995 ts_manager.cc:194] Registered new tserver with Master: 359653970fde48208e10754a65c44608 (127.0.97.2:36025)
I20260620 02:59:14.835175 3995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60915
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:59:14.861466 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34247
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:15.180097 4323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:15.180380 4323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:15.180547 4323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:15.187898 4323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:15.188067 4323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:15.188257 4323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:59:15.198050 4323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:15.199337 4323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:15.200664 4323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:15.214193 4328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:15.214241 4331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:15.214278 4329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:15.215004 4323 server_base.cc:1034] running on GCE node
I20260620 02:59:15.215606 4323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:15.216495 4323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:15.217780 4323 hybrid_clock.cc:648] HybridClock initialized: now 1781924355217690 us; error 85 us; skew 500 ppm
I20260620 02:59:15.218111 4323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:15 dist-test-slave-f1j0 krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:15.221239 4323 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:59:15.222536 4323 webserver.cc:460] Webserver started at http://127.0.97.3:45917/ using document root <none> and password file <none>
I20260620 02:59:15.223166 4323 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:15.223261 4323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:15.223500 4323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:15.226305 4323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "fc25c64362cc4c3dae17bd4af3f2b846"
format_stamp: "Formatted at 2026-06-20 02:59:15 on dist-test-slave-f1j0"
server_key: "6c450a2df17dcc77c592381c91126459"
server_key_iv: "5bb76587bd9b9bab8ca909875810d8bf"
server_key_version: "encryptionkey@0"
I20260620 02:59:15.226994 4323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fc25c64362cc4c3dae17bd4af3f2b846"
format_stamp: "Formatted at 2026-06-20 02:59:15 on dist-test-slave-f1j0"
server_key: "6c450a2df17dcc77c592381c91126459"
server_key_iv: "5bb76587bd9b9bab8ca909875810d8bf"
server_key_version: "encryptionkey@0"
I20260620 02:59:15.231896 4323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:59:15.235327 4338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:15.236631 4323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:59:15.236774 4323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "fc25c64362cc4c3dae17bd4af3f2b846"
format_stamp: "Formatted at 2026-06-20 02:59:15 on dist-test-slave-f1j0"
server_key: "6c450a2df17dcc77c592381c91126459"
server_key_iv: "5bb76587bd9b9bab8ca909875810d8bf"
server_key_version: "encryptionkey@0"
I20260620 02:59:15.236943 4323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:15.254526 4323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:15.258846 4323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:15.259083 4323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:15.259959 4323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:15.261168 4323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:15.261240 4323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:15.261332 4323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:15.261374 4323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:15.293304 4323 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39231
I20260620 02:59:15.293396 4450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39231 every 8 connection(s)
I20260620 02:59:15.294603 4323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:59:15.299036 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 4323
I20260620 02:59:15.299213 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:59:15.299749 388 external_mini_cluster.cc:1428] Setting key 466f2007db57e65defb81236bb384e73
Jun 20 02:59:15 dist-test-slave-f1j0 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:15.315320 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:15.297321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33139 (local address 127.0.97.62:34247)
0620 02:59:15.297528 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:15.297534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:15.298901 (+ 1367us) server_negotiation.cc:407] Connection header received
0620 02:59:15.301345 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:15.301353 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:15.301437 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:15.301545 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:15.303224 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:15.304004 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:15.305436 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:15.305924 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:15.306015 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:15.309264 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:15.309269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:15.309273 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:15.309312 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:15.312133 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:15.312806 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:15.312813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:15.312816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:15.312971 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:15.313449 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:15.313453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:15.313456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:15.313866 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:15.314019 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:15.314748 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 02:59:15.314958 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:59:15.316349 4453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:15.297894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34247 (local address 127.0.97.3:33139)
0620 02:59:15.298734 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:15.298791 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:15.301066 (+ 2275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:15.301814 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:15.301822 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:15.302469 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:15.303077 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:15.303095 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:15.304137 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:15.304141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:15.305320 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:15.305331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:15.306123 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:15.306127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:15.306333 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:15.306932 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:15.306952 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:15.309074 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:15.312242 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:15.312248 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:15.312260 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:15.312633 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:15.313105 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:15.313109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:15.313112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:15.313341 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:15.313986 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:15.313993 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:15.314372 (+ 379us) client_negotiation.cc:769] Sending connection context
0620 02:59:15.314640 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 02:59:15.315162 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":312,"threads_started":1}
I20260620 02:59:15.317776 4451 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34247
I20260620 02:59:15.318056 4451 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:15.318743 4451 heartbeater.cc:511] Master 127.0.97.62:34247 requested a full tablet report, sending...
I20260620 02:59:15.320101 3995 ts_manager.cc:194] Registered new tserver with Master: fc25c64362cc4c3dae17bd4af3f2b846 (127.0.97.3:39231)
I20260620 02:59:15.320904 3995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33139
I20260620 02:59:15.327277 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:59:15.330668 4184 heartbeater.cc:503] Master 127.0.97.62:34247 was elected leader, sending a full tablet report...
Jun 20 02:59:15 dist-test-slave-f1j0 krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:59:15 dist-test-slave-f1j0 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:59:15.374639 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:15.359950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58820 (local address 127.0.97.1:45255)
0620 02:59:15.360627 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:15.360639 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:15.360675 (+ 36us) server_negotiation.cc:407] Connection header received
0620 02:59:15.360756 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:15.360762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:15.360980 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:15.361148 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:15.362577 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:15.363557 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:15.364776 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:15.365330 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:15.365473 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:15.369191 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:15.369201 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:15.369215 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:15.369247 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:15.371828 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:15.372449 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:15.372456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:15.372458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:15.372570 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:15.372922 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:15.372929 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:15.372932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:15.373420 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:15.373752 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:15.374043 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 02:59:15.374293 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":349,"threads_started":1}
Jun 20 02:59:15 dist-test-slave-f1j0 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:15.391809 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:15.378635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.0.97.62:34247)
0620 02:59:15.378851 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:15.378860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:15.379103 (+ 243us) server_negotiation.cc:407] Connection header received
0620 02:59:15.379291 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:15.379296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:15.379369 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:15.379467 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:15.380603 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:15.381317 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:15.382585 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:15.382995 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:15.383117 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:15.387013 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:15.387018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:15.387022 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:15.387054 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:15.389296 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:15.389943 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:15.389950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:15.389955 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:15.390071 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:15.390420 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:15.390425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:15.390428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:15.390735 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:15.390970 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:15.391283 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 02:59:15.391507 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 02:59:15.392381 3995 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56714
I20260620 02:59:15.393358 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 4056
I20260620 02:59:15.405746 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:15.394237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58834 (local address 127.0.97.1:45255)
0620 02:59:15.394458 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:15.394466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:15.394669 (+ 203us) server_negotiation.cc:407] Connection header received
0620 02:59:15.394846 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:15.394851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:15.394921 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:15.395019 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:15.396119 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:15.396789 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:15.398175 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:15.398606 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:15.398750 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:15.400737 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:15.400742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:15.400745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:15.400787 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:15.403261 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:15.403869 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:15.403874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:15.403884 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:15.404026 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:15.404391 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:15.404397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:15.404400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:15.404632 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:15.404906 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:15.405173 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 02:59:15.405377 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 02:59:15.406327 4179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:15.489936 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 4056
I20260620 02:59:15.505966 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 4190
Jun 20 02:59:15 dist-test-slave-f1j0 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:59:15.521783 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:15.506855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.0.97.2:36025)
0620 02:59:15.507366 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:15.507373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:15.507398 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:59:15.507455 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:15.507462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:15.507675 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:15.507800 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:15.508918 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:15.509951 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:15.511231 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:15.511771 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:15.511867 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:15.515808 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:15.515819 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:15.515834 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:15.515866 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:15.518887 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:15.519632 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:15.519640 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:15.519642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:15.519759 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:15.520124 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:15.520131 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:15.520134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:15.520471 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:15.520795 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:15.521126 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 02:59:15.521387 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":256,"threads_started":1}
I20260620 02:59:15.523030 4313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:15.604815 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 4190
I20260620 02:59:15.621042 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 4323
Jun 20 02:59:15 dist-test-slave-f1j0 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:59:15.637029 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:15.622068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55276 (local address 127.0.97.3:39231)
0620 02:59:15.622655 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:15.622662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:15.622690 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:59:15.622752 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:15.622758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:15.622959 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:15.623139 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:15.624353 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:15.625364 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:15.626799 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:15.627371 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:15.627507 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:15.631645 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:15.631656 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:15.631669 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:15.631703 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:15.634318 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:15.634995 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:15.635002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:15.635004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:15.635119 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:15.635479 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:15.635486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:15.635489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:15.635824 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:15.636135 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:15.636436 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 02:59:15.636691 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":306,"threads_started":1}
I20260620 02:59:15.638356 4446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:15.720538 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 4323
I20260620 02:59:15.736330 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 3963
I20260620 02:59:15.737473 4025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:15.837683 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 3963
2026-06-20T02:59:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3871 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:59:15 dist-test-slave-f1j0 krb5kdc[4470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:59:15 dist-test-slave-f1j0 krb5kdc[4470](info): set up 2 sockets
Jun 20 02:59:15 dist-test-slave-f1j0 krb5kdc[4470](info): commencing operation
krb5kdc: starting...
W20260620 02:59:16.922168 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 02:59:17 dist-test-slave-f1j0 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:59:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:59:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:59:17.119558 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43639
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:17.433418 4483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:17.433774 4483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:17.433877 4483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:17.441009 4483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:59:17.441116 4483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:17.441164 4483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:17.441220 4483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:59:17.441267 4483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:59:17.451078 4483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:17.452308 4483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:17.453542 4483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:17.464969 4488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:17.464931 4491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:17.464973 4489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:17.465693 4483 server_base.cc:1034] running on GCE node
I20260620 02:59:17.466450 4483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:17.467877 4483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:17.469090 4483 hybrid_clock.cc:648] HybridClock initialized: now 1781924357469040 us; error 58 us; skew 500 ppm
I20260620 02:59:17.469456 4483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:17 dist-test-slave-f1j0 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:17.472484 4483 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:59:17.473631 4483 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:17.473763 4483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:17.474080 4483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:17.476979 4483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "c29c49b95f34447fafb35d2182667cd5"
format_stamp: "Formatted at 2026-06-20 02:59:17 on dist-test-slave-f1j0"
server_key: "52fda252764c3181688736c9922d6703"
server_key_iv: "c43fdb356f43c2f9de83bd41e05381cb"
server_key_version: "encryptionkey@0"
I20260620 02:59:17.477842 4483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "c29c49b95f34447fafb35d2182667cd5"
format_stamp: "Formatted at 2026-06-20 02:59:17 on dist-test-slave-f1j0"
server_key: "52fda252764c3181688736c9922d6703"
server_key_iv: "c43fdb356f43c2f9de83bd41e05381cb"
server_key_version: "encryptionkey@0"
I20260620 02:59:17.482720 4483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 02:59:17.485953 4497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:17.487251 4483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:59:17.487419 4483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "c29c49b95f34447fafb35d2182667cd5"
format_stamp: "Formatted at 2026-06-20 02:59:17 on dist-test-slave-f1j0"
server_key: "52fda252764c3181688736c9922d6703"
server_key_iv: "c43fdb356f43c2f9de83bd41e05381cb"
server_key_version: "encryptionkey@0"
I20260620 02:59:17.487641 4483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:17.500734 4483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:17.504680 4483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:17.504957 4483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:17.522747 4483 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43639
I20260620 02:59:17.522724 4548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43639 every 8 connection(s)
I20260620 02:59:17.523654 4483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:59:17.527263 4549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:17.528548 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 4483
I20260620 02:59:17.528754 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 02:59:17.529639 388 external_mini_cluster.cc:1428] Setting key 78d788785c661bab42ad1ce3b8074d29
I20260620 02:59:17.541654 4549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5: Bootstrap starting.
Jun 20 02:59:17 dist-test-slave-f1j0 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:17.545253 4549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:17.546458 4549 log.cc:824] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:17.549194 4549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5: No bootstrap required, opened a new log
I20260620 02:59:17.551774 4552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:17.531624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58758 (local address 127.0.97.62:43639)
0620 02:59:17.532402 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:17.532413 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:17.532454 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:59:17.534006 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:17.534027 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:17.534409 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:17.534734 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:17.536448 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:17.537773 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:17.539584 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:17.540238 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:17.540364 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:17.544822 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:17.544830 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:17.544845 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:17.544879 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:17.547889 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:17.548524 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:17.548540 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:17.548546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:17.548698 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:17.549062 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:17.549068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:17.549072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:17.549689 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:17.549996 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:17.550404 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 02:59:17.550799 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":372,"threads_started":1}
I20260620 02:59:17.557013 4549 raft_consensus.cc:348] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29c49b95f34447fafb35d2182667cd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43639 } }
I20260620 02:59:17.557355 4549 raft_consensus.cc:374] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:17.557457 4549 raft_consensus.cc:729] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c29c49b95f34447fafb35d2182667cd5, State: Initialized, Role: FOLLOWER
I20260620 02:59:17.558079 4549 consensus_queue.cc:260] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29c49b95f34447fafb35d2182667cd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43639 } }
I20260620 02:59:17.558365 4549 raft_consensus.cc:388] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:59:17.558517 4549 raft_consensus.cc:482] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:59:17.558714 4549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:17.560896 4549 raft_consensus.cc:504] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29c49b95f34447fafb35d2182667cd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43639 } }
I20260620 02:59:17.561398 4549 leader_election.cc:302] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c29c49b95f34447fafb35d2182667cd5; no voters:
I20260620 02:59:17.562104 4549 leader_election.cc:288] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:59:17.562289 4554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:17.562979 4554 raft_consensus.cc:686] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5 [term 1 LEADER]: Becoming Leader. State: Replica: c29c49b95f34447fafb35d2182667cd5, State: Running, Role: LEADER
I20260620 02:59:17.563481 4554 consensus_queue.cc:237] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29c49b95f34447fafb35d2182667cd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43639 } }
I20260620 02:59:17.564133 4549 sys_catalog.cc:564] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:59:17.566196 4556 sys_catalog.cc:455] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c29c49b95f34447fafb35d2182667cd5. Latest consensus state: current_term: 1 leader_uuid: "c29c49b95f34447fafb35d2182667cd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29c49b95f34447fafb35d2182667cd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43639 } } }
I20260620 02:59:17.566645 4556 sys_catalog.cc:458] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:17.566280 4555 sys_catalog.cc:455] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c29c49b95f34447fafb35d2182667cd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29c49b95f34447fafb35d2182667cd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43639 } } }
I20260620 02:59:17.566874 4555 sys_catalog.cc:458] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:17.568428 4563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:59:17.573624 4563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:59:17.582070 4563 catalog_manager.cc:1347] Generated new cluster ID: ce46e99da45c4c5bbc7934e6de6d81da
I20260620 02:59:17.582297 4563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:59:17.618008 4563 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:59:17.619513 4563 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:59:17.631060 4563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5: Generated new TSK 0
I20260620 02:59:17.632121 4563 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:59:17.695218 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43639
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:18.045403 4575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:18.045677 4575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:18.045883 4575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:18.052938 4575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:18.053036 4575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:18.053246 4575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:59:18.062889 4575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:18.064213 4575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:18.065845 4575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:18.079811 4581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:18.079919 4583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:18.079998 4580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:18.081121 4575 server_base.cc:1034] running on GCE node
I20260620 02:59:18.081843 4575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:18.082896 4575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:18.084124 4575 hybrid_clock.cc:648] HybridClock initialized: now 1781924358084065 us; error 56 us; skew 500 ppm
I20260620 02:59:18.084447 4575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:18 dist-test-slave-f1j0 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:18.087519 4575 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:59:18.088796 4575 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:18.088929 4575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:18.089242 4575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:18.092341 4575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "2ab1de495f4d4c66b79b779ce6d8b26e"
format_stamp: "Formatted at 2026-06-20 02:59:18 on dist-test-slave-f1j0"
server_key: "3d648d91fd57b5ca801c8fe25ee5f59c"
server_key_iv: "b0ff8ef3ea1a2bc6ea33fe4891bce4e6"
server_key_version: "encryptionkey@0"
I20260620 02:59:18.093154 4575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2ab1de495f4d4c66b79b779ce6d8b26e"
format_stamp: "Formatted at 2026-06-20 02:59:18 on dist-test-slave-f1j0"
server_key: "3d648d91fd57b5ca801c8fe25ee5f59c"
server_key_iv: "b0ff8ef3ea1a2bc6ea33fe4891bce4e6"
server_key_version: "encryptionkey@0"
I20260620 02:59:18.098153 4575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:59:18.101694 4589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:18.103130 4575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:59:18.103309 4575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "2ab1de495f4d4c66b79b779ce6d8b26e"
format_stamp: "Formatted at 2026-06-20 02:59:18 on dist-test-slave-f1j0"
server_key: "3d648d91fd57b5ca801c8fe25ee5f59c"
server_key_iv: "b0ff8ef3ea1a2bc6ea33fe4891bce4e6"
server_key_version: "encryptionkey@0"
I20260620 02:59:18.103521 4575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:18.126750 4575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:18.131008 4575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:18.131309 4575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:18.132305 4575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:18.133793 4575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:18.133909 4575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:18.134039 4575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:18.134141 4575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:18.167815 4575 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41029
I20260620 02:59:18.167897 4701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41029 every 8 connection(s)
I20260620 02:59:18.168707 4575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:59:18.177412 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 4575
I20260620 02:59:18.177660 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:59:18.178325 388 external_mini_cluster.cc:1428] Setting key 174ea7bbd77d9fe0aa36a5c874cfdfb6
Jun 20 02:59:18 dist-test-slave-f1j0 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:18.189672 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:18.171199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59771 (local address 127.0.97.62:43639)
0620 02:59:18.171649 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:18.171656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:18.172634 (+ 978us) server_negotiation.cc:407] Connection header received
0620 02:59:18.174651 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:18.174656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:18.174724 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:18.174838 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:18.176449 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:18.177387 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:18.179486 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:18.179936 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:18.180083 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:18.183651 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:18.183656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:18.183659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:18.183698 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:18.186701 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:18.187437 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:18.187441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:18.187444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:18.187549 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:18.187955 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:18.187959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:18.187962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:18.188254 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:18.188407 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:18.189089 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 02:59:18.189297 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":209,"threads_started":1}
I20260620 02:59:18.190691 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:18.171760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43639 (local address 127.0.97.1:59771)
0620 02:59:18.172470 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:18.172513 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:18.174365 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:18.175047 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:18.175056 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:18.175612 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:18.176233 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:18.176251 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:18.177626 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:18.177629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:18.179218 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:18.179231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:18.180228 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:18.180236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:18.180477 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:18.181167 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:18.181198 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:18.183414 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:18.186838 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:18.186848 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:18.186873 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:18.187296 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:18.187657 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:18.187662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:18.187665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:18.187838 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:18.188388 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:18.188396 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:18.188725 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 02:59:18.188985 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 02:59:18.189532 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":279,"threads_started":1}
I20260620 02:59:18.192340 4702 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43639
I20260620 02:59:18.192647 4702 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:18.193287 4702 heartbeater.cc:511] Master 127.0.97.62:43639 requested a full tablet report, sending...
I20260620 02:59:18.195428 4514 ts_manager.cc:194] Registered new tserver with Master: 2ab1de495f4d4c66b79b779ce6d8b26e (127.0.97.1:41029)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:59:18.196820 4514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:59:18.221297 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43639
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:18.542171 4708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:18.542452 4708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:18.542553 4708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:18.549906 4708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:18.550012 4708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:18.550140 4708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:59:18.559722 4708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:18.560930 4708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:18.562247 4708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:18.575062 4713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:18.575021 4714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:18.575609 4716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:18.575891 4708 server_base.cc:1034] running on GCE node
I20260620 02:59:18.576529 4708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:18.577389 4708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:18.578644 4708 hybrid_clock.cc:648] HybridClock initialized: now 1781924358578595 us; error 49 us; skew 500 ppm
I20260620 02:59:18.578959 4708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:18 dist-test-slave-f1j0 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:18.582168 4708 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:59:18.583320 4708 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:18.583451 4708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:18.583770 4708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:18.586613 4708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "ea42eb67bb0a4b99858a55d8c47da207"
format_stamp: "Formatted at 2026-06-20 02:59:18 on dist-test-slave-f1j0"
server_key: "8146de54aedce177b086ebe6c4b89592"
server_key_iv: "3d4f74d9b041d5bff50217fb9a4cafbd"
server_key_version: "encryptionkey@0"
I20260620 02:59:18.587425 4708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ea42eb67bb0a4b99858a55d8c47da207"
format_stamp: "Formatted at 2026-06-20 02:59:18 on dist-test-slave-f1j0"
server_key: "8146de54aedce177b086ebe6c4b89592"
server_key_iv: "3d4f74d9b041d5bff50217fb9a4cafbd"
server_key_version: "encryptionkey@0"
I20260620 02:59:18.592458 4708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:59:18.595984 4722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:18.597335 4708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 02:59:18.597515 4708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "ea42eb67bb0a4b99858a55d8c47da207"
format_stamp: "Formatted at 2026-06-20 02:59:18 on dist-test-slave-f1j0"
server_key: "8146de54aedce177b086ebe6c4b89592"
server_key_iv: "3d4f74d9b041d5bff50217fb9a4cafbd"
server_key_version: "encryptionkey@0"
I20260620 02:59:18.597788 4708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:18.613363 4708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:18.617415 4708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:18.617719 4708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:18.618661 4708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:18.620044 4708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:18.620128 4708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:18.620273 4708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:18.620349 4708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:18.653115 4708 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43537
I20260620 02:59:18.653221 4834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43537 every 8 connection(s)
I20260620 02:59:18.654034 4708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:59:18.659905 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 4708
I20260620 02:59:18.660089 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:59:18.660504 388 external_mini_cluster.cc:1428] Setting key ab6cf47e84f6cb5d9aacc1ccee92bfb8
Jun 20 02:59:18 dist-test-slave-f1j0 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:18.675333 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:18.656487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59347 (local address 127.0.97.62:43639)
0620 02:59:18.656667 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:18.656672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:18.657822 (+ 1150us) server_negotiation.cc:407] Connection header received
0620 02:59:18.659605 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:18.659611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:18.659709 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:18.659818 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:18.661479 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:18.662349 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:18.663871 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:18.664639 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:18.664774 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:18.668560 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:18.668567 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:18.668574 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:18.668619 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:18.672142 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:18.672833 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:18.672841 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:18.672848 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:18.672993 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:18.673375 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:18.673383 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:18.673389 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:18.673801 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:18.673949 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:18.674610 (+ 661us) server_negotiation.cc:299] Negotiation successful
0620 02:59:18.674888 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:59:18.676167 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:18.656894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43639 (local address 127.0.97.2:59347)
0620 02:59:18.657657 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:18.657703 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:18.659340 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:18.660063 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:18.660076 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:18.660672 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:18.661293 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:18.661321 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:18.662507 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:18.662511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:18.663725 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:18.663736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:18.664830 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:18.664835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:18.665010 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:18.665763 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:18.665785 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:18.668335 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:18.672276 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:18.672283 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:18.672295 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:18.672717 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:18.673103 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:18.673106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:18.673108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:18.673266 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:18.673921 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:18.673927 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:18.674251 (+ 324us) client_negotiation.cc:769] Sending connection context
0620 02:59:18.674482 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 02:59:18.674989 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":293,"threads_started":1}
I20260620 02:59:18.677893 4835 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43639
I20260620 02:59:18.678272 4835 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:18.678964 4835 heartbeater.cc:511] Master 127.0.97.62:43639 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:59:18.680604 4514 ts_manager.cc:194] Registered new tserver with Master: ea42eb67bb0a4b99858a55d8c47da207 (127.0.97.2:43537)
I20260620 02:59:18.681459 4514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:59:18.705192 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43639
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:19.024055 4840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:19.024407 4840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:19.024587 4840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:19.031793 4840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:19.031962 4840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:19.032169 4840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:59:19.041833 4840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:19.043238 4840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:19.044552 4840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:19.057639 4846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:19.057870 4848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:19.057668 4845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:19.058290 4840 server_base.cc:1034] running on GCE node
I20260620 02:59:19.058880 4840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:19.059755 4840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:19.060976 4840 hybrid_clock.cc:648] HybridClock initialized: now 1781924359060917 us; error 57 us; skew 500 ppm
I20260620 02:59:19.061290 4840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:19 dist-test-slave-f1j0 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:19.064424 4840 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:59:19.065760 4840 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:19.065893 4840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:19.066202 4840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:19.069499 4840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "ea29ed00dd174fa19146cbce97624278"
format_stamp: "Formatted at 2026-06-20 02:59:19 on dist-test-slave-f1j0"
server_key: "9a866e30383fd65040a3a79ff6bc3df3"
server_key_iv: "a83a135f2ae3ebcc9b5136236871a33e"
server_key_version: "encryptionkey@0"
I20260620 02:59:19.070627 4840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ea29ed00dd174fa19146cbce97624278"
format_stamp: "Formatted at 2026-06-20 02:59:19 on dist-test-slave-f1j0"
server_key: "9a866e30383fd65040a3a79ff6bc3df3"
server_key_iv: "a83a135f2ae3ebcc9b5136236871a33e"
server_key_version: "encryptionkey@0"
I20260620 02:59:19.076694 4840 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:59:19.080111 4854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:19.081502 4840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:59:19.081705 4840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "ea29ed00dd174fa19146cbce97624278"
format_stamp: "Formatted at 2026-06-20 02:59:19 on dist-test-slave-f1j0"
server_key: "9a866e30383fd65040a3a79ff6bc3df3"
server_key_iv: "a83a135f2ae3ebcc9b5136236871a33e"
server_key_version: "encryptionkey@0"
I20260620 02:59:19.081928 4840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:19.115351 4840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:19.119700 4840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:19.119982 4840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:19.120993 4840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:19.122447 4840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:19.122529 4840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:19.122694 4840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:19.122772 4840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:19.156308 4840 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46871
I20260620 02:59:19.156322 4966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46871 every 8 connection(s)
I20260620 02:59:19.157238 4840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:59:19.163847 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 4840
I20260620 02:59:19.164034 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:59:19.164486 388 external_mini_cluster.cc:1428] Setting key b0ac441a1215fc7a6a898db5dc9617d9
Jun 20 02:59:19 dist-test-slave-f1j0 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:19.177446 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:19.159822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42205 (local address 127.0.97.62:43639)
0620 02:59:19.160008 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:19.160012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:19.161109 (+ 1097us) server_negotiation.cc:407] Connection header received
0620 02:59:19.162897 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:19.162901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:19.162969 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:19.163064 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:19.164764 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.165757 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.167699 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.168434 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.168578 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:19.171983 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:19.171991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:19.171999 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:19.172040 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:19.174737 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:19.175404 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:19.175407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:19.175409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:19.175534 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:19.175984 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:19.175988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:19.175990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:19.176275 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:19.176398 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:19.177044 (+ 646us) server_negotiation.cc:299] Negotiation successful
0620 02:59:19.177207 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:59:19.178530 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:19.160264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43639 (local address 127.0.97.3:42205)
0620 02:59:19.160940 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:19.160978 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:19.162632 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:19.163253 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:19.163261 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:19.163847 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:19.164538 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:19.164554 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.165908 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.165913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:19.167527 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:19.167545 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.168630 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.168635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:19.168814 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:19.169477 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:19.169498 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:19.171793 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:19.174852 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:19.174859 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:19.174870 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:19.175273 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:19.175642 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:19.175646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:19.175648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:19.175839 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:19.176390 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:19.176395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:19.176676 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 02:59:19.176909 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 02:59:19.177270 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":269,"threads_started":1}
I20260620 02:59:19.179932 4967 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43639
I20260620 02:59:19.180233 4967 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:19.180828 4967 heartbeater.cc:511] Master 127.0.97.62:43639 requested a full tablet report, sending...
I20260620 02:59:19.182236 4514 ts_manager.cc:194] Registered new tserver with Master: ea29ed00dd174fa19146cbce97624278 (127.0.97.3:46871)
I20260620 02:59:19.183029 4514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42205
I20260620 02:59:19.192059 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:59:19.200021 4702 heartbeater.cc:503] Master 127.0.97.62:43639 was elected leader, sending a full tablet report...
I20260620 02:59:19.209012 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:19.196673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.0.97.62:43639)
0620 02:59:19.196900 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:19.196909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:19.197299 (+ 390us) server_negotiation.cc:407] Connection header received
0620 02:59:19.197421 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:19.197424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:19.197480 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:19.197556 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:19.198998 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.199664 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.201001 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.201684 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.201856 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:19.204081 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:19.204087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:19.204092 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:19.204131 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:19.206416 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:19.207044 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:19.207051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:19.207054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:19.207229 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:19.207579 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:19.207584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:19.207588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:19.207882 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:19.208142 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:19.208476 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 02:59:19.208700 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:59:19.212777 4514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:59:19.219367 4514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:59:19.256623 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:19.231236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38832 (local address 127.0.97.1:41029)
0620 02:59:19.243934 (+ 12698us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:19.243948 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:19.246559 (+ 2611us) server_negotiation.cc:407] Connection header received
0620 02:59:19.247802 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:19.247816 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:19.248150 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:19.248348 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:19.249434 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.250994 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.254051 (+ 3057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.255122 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.255241 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:19.255477 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:19.256126 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 02:59:19.256305 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12456,"thread_start_us":12740,"threads_started":1}
I20260620 02:59:19.256788 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:19.231078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41029 (local address 127.0.0.1:38832)
0620 02:59:19.246299 (+ 15221us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:19.246379 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:19.247529 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:19.248708 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:19.248715 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:19.248743 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:19.249244 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:19.249256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.251138 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.251144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:19.253913 (+ 2769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:19.253921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.255623 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.255635 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:19.255861 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:19.255917 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 02:59:19.256071 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 02:59:19.256282 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14833,"mutex_wait_us":571,"thread_start_us":15529,"threads_started":1}
I20260620 02:59:19.256840 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:19.231157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43537 (local address 127.0.0.1:42052)
0620 02:59:19.247091 (+ 15934us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:19.247115 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:19.247333 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:19.248417 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:19.248425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:19.248462 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:19.249038 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:19.249049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.251055 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.251064 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:19.253670 (+ 2606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:19.253684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.255705 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.255711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:19.256019 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:19.256171 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 02:59:19.256254 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 02:59:19.256399 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15821,"mutex_wait_us":194,"thread_start_us":17454,"threads_started":1}
I20260620 02:59:19.256896 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:19.232894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.0.97.3:46871)
0620 02:59:19.246283 (+ 13389us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:19.246292 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:19.249083 (+ 2791us) server_negotiation.cc:407] Connection header received
0620 02:59:19.249218 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:19.249226 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:19.249466 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:19.249681 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:19.250200 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.251711 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.253723 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.255219 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.255343 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:19.255566 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:19.256375 (+ 809us) server_negotiation.cc:299] Negotiation successful
0620 02:59:19.256555 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13200,"thread_start_us":13023,"threads_started":1}
I20260620 02:59:19.257537 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:19.232415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42052 (local address 127.0.97.2:43537)
0620 02:59:19.246658 (+ 14243us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:19.246666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:19.247541 (+ 875us) server_negotiation.cc:407] Connection header received
0620 02:59:19.247650 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:19.247660 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:19.247936 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:19.248157 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:19.249236 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.250890 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.253830 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.255219 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.255343 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:19.255522 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:19.257201 (+ 1679us) server_negotiation.cc:299] Negotiation successful
0620 02:59:19.257319 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14119,"thread_start_us":14017,"threads_started":1}
I20260620 02:59:19.258137 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:19.231629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46871 (local address 127.0.0.1:51894)
0620 02:59:19.248927 (+ 17298us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:19.248950 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:19.249080 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:19.249678 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:19.249683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:19.249703 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:19.250027 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:19.250042 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.251859 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.251866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:19.253502 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:19.253513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.255719 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.255725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:19.256019 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:19.256168 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 02:59:19.256254 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 02:59:19.257820 (+ 1566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17139,"mutex_wait_us":1403,"thread_start_us":15214,"threads_started":1}
I20260620 02:59:19.261051 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 5d603c3e0e734c77b397eb00adef0c1b (DEFAULT_TABLE table=test-table [id=183153c9a1084e249ffd431710a5c7ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:19.261102 4770 tablet_service.cc:1467] Processing CreateTablet for tablet 5d603c3e0e734c77b397eb00adef0c1b (DEFAULT_TABLE table=test-table [id=183153c9a1084e249ffd431710a5c7ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:19.261255 4902 tablet_service.cc:1467] Processing CreateTablet for tablet 5d603c3e0e734c77b397eb00adef0c1b (DEFAULT_TABLE table=test-table [id=183153c9a1084e249ffd431710a5c7ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:19.262311 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d603c3e0e734c77b397eb00adef0c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:19.262456 4902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d603c3e0e734c77b397eb00adef0c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:19.262485 4770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d603c3e0e734c77b397eb00adef0c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:19.274011 4985 tablet_bootstrap.cc:492] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e: Bootstrap starting.
I20260620 02:59:19.276836 4987 tablet_bootstrap.cc:492] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278: Bootstrap starting.
I20260620 02:59:19.278307 4985 tablet_bootstrap.cc:654] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:19.279788 4985 log.cc:824] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:19.280268 4987 tablet_bootstrap.cc:654] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:19.281323 4986 tablet_bootstrap.cc:492] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207: Bootstrap starting.
I20260620 02:59:19.281718 4987 log.cc:824] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:19.283433 4985 tablet_bootstrap.cc:492] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e: No bootstrap required, opened a new log
I20260620 02:59:19.283702 4985 ts_tablet_manager.cc:1397] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260620 02:59:19.284552 4987 tablet_bootstrap.cc:492] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278: No bootstrap required, opened a new log
I20260620 02:59:19.284809 4987 ts_tablet_manager.cc:1397] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 02:59:19.288763 4986 tablet_bootstrap.cc:654] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:19.289808 4986 log.cc:824] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:19.292656 4986 tablet_bootstrap.cc:492] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207: No bootstrap required, opened a new log
I20260620 02:59:19.292910 4986 ts_tablet_manager.cc:1397] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260620 02:59:19.294260 4985 raft_consensus.cc:348] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab1de495f4d4c66b79b779ce6d8b26e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41029 } } peers { permanent_uuid: "ea42eb67bb0a4b99858a55d8c47da207" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43537 } } peers { permanent_uuid: "ea29ed00dd174fa19146cbce97624278" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46871 } }
I20260620 02:59:19.294675 4985 raft_consensus.cc:374] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:19.294770 4985 raft_consensus.cc:729] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab1de495f4d4c66b79b779ce6d8b26e, State: Initialized, Role: FOLLOWER
I20260620 02:59:19.295387 4985 consensus_queue.cc:260] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab1de495f4d4c66b79b779ce6d8b26e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41029 } } peers { permanent_uuid: "ea42eb67bb0a4b99858a55d8c47da207" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43537 } } peers { permanent_uuid: "ea29ed00dd174fa19146cbce97624278" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46871 } }
I20260620 02:59:19.295329 4987 raft_consensus.cc:348] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab1de495f4d4c66b79b779ce6d8b26e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41029 } } peers { permanent_uuid: "ea42eb67bb0a4b99858a55d8c47da207" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43537 } } peers { permanent_uuid: "ea29ed00dd174fa19146cbce97624278" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46871 } }
I20260620 02:59:19.295764 4987 raft_consensus.cc:374] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:19.295902 4987 raft_consensus.cc:729] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea29ed00dd174fa19146cbce97624278, State: Initialized, Role: FOLLOWER
I20260620 02:59:19.297178 4985 ts_tablet_manager.cc:1428] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 02:59:19.297667 4987 consensus_queue.cc:260] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab1de495f4d4c66b79b779ce6d8b26e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41029 } } peers { permanent_uuid: "ea42eb67bb0a4b99858a55d8c47da207" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43537 } } peers { permanent_uuid: "ea29ed00dd174fa19146cbce97624278" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46871 } }
I20260620 02:59:19.301699 4986 raft_consensus.cc:348] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab1de495f4d4c66b79b779ce6d8b26e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41029 } } peers { permanent_uuid: "ea42eb67bb0a4b99858a55d8c47da207" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43537 } } peers { permanent_uuid: "ea29ed00dd174fa19146cbce97624278" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46871 } }
I20260620 02:59:19.302096 4986 raft_consensus.cc:374] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:19.302191 4986 raft_consensus.cc:729] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea42eb67bb0a4b99858a55d8c47da207, State: Initialized, Role: FOLLOWER
I20260620 02:59:19.302235 4967 heartbeater.cc:503] Master 127.0.97.62:43639 was elected leader, sending a full tablet report...
I20260620 02:59:19.302744 4987 ts_tablet_manager.cc:1428] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278: Time spent starting tablet: real 0.018s user 0.014s sys 0.002s
I20260620 02:59:19.302732 4986 consensus_queue.cc:260] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab1de495f4d4c66b79b779ce6d8b26e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41029 } } peers { permanent_uuid: "ea42eb67bb0a4b99858a55d8c47da207" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43537 } } peers { permanent_uuid: "ea29ed00dd174fa19146cbce97624278" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46871 } }
I20260620 02:59:19.303838 4835 heartbeater.cc:503] Master 127.0.97.62:43639 was elected leader, sending a full tablet report...
I20260620 02:59:19.304260 4986 ts_tablet_manager.cc:1428] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260620 02:59:19.405861 4836 tablet.cc:2369] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:59:19.408694 4968 tablet.cc:2369] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:59:19.411702 4993 raft_consensus.cc:482] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:59:19.412004 4993 raft_consensus.cc:504] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab1de495f4d4c66b79b779ce6d8b26e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41029 } } peers { permanent_uuid: "ea42eb67bb0a4b99858a55d8c47da207" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43537 } } peers { permanent_uuid: "ea29ed00dd174fa19146cbce97624278" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46871 } }
I20260620 02:59:19.413784 4993 leader_election.cc:288] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ab1de495f4d4c66b79b779ce6d8b26e (127.0.97.1:41029), ea29ed00dd174fa19146cbce97624278 (127.0.97.3:46871)
I20260620 02:59:19.420259 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:19.414127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48141 (local address 127.0.97.1:41029)
0620 02:59:19.414325 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:19.414332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:19.415034 (+ 702us) server_negotiation.cc:407] Connection header received
0620 02:59:19.415265 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:19.415269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:19.415326 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:19.415420 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:19.415958 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.416671 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.418580 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.419300 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.419450 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:19.419561 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:19.419877 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 02:59:19.419990 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:59:19.420455 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:19.414075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41029 (local address 127.0.97.2:48141)
0620 02:59:19.414911 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:19.414930 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:19.415112 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:19.415436 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:19.415440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:19.415462 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:19.415809 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:19.415818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.416798 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.416805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:19.418421 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:19.418432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.419539 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.419546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:19.419698 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:19.419755 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 02:59:19.419853 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 02:59:19.419973 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":117,"thread_start_us":428,"threads_started":1}
I20260620 02:59:19.420907 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:19.414264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41323 (local address 127.0.97.3:46871)
0620 02:59:19.414503 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:19.414511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:19.415140 (+ 629us) server_negotiation.cc:407] Connection header received
0620 02:59:19.415333 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:19.415337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:19.415393 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:19.415483 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:19.416149 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.416833 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.418637 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.419365 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.419506 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:19.419620 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:19.419925 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 02:59:19.420039 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 02:59:19.420955 4703 tablet.cc:2369] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:59:19.421414 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:19.414270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46871 (local address 127.0.97.2:41323)
0620 02:59:19.415016 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:19.415030 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:19.415195 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:19.415507 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:19.415514 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:19.415578 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:19.415993 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:19.416002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.416966 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.416972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:19.418481 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:19.418491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.419571 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.419576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:19.419787 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:19.419808 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 02:59:19.419903 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 02:59:19.420023 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":48,"spinlock_wait_cycles":25984,"thread_start_us":432,"threads_started":1}
I20260620 02:59:19.422297 4657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d603c3e0e734c77b397eb00adef0c1b" candidate_uuid: "ea42eb67bb0a4b99858a55d8c47da207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab1de495f4d4c66b79b779ce6d8b26e" is_pre_election: true
I20260620 02:59:19.422379 4922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d603c3e0e734c77b397eb00adef0c1b" candidate_uuid: "ea42eb67bb0a4b99858a55d8c47da207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea29ed00dd174fa19146cbce97624278" is_pre_election: true
I20260620 02:59:19.422813 4657 raft_consensus.cc:2454] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea42eb67bb0a4b99858a55d8c47da207 in term 0.
I20260620 02:59:19.422842 4922 raft_consensus.cc:2454] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea42eb67bb0a4b99858a55d8c47da207 in term 0.
I20260620 02:59:19.423468 4723 leader_election.cc:302] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2ab1de495f4d4c66b79b779ce6d8b26e, ea29ed00dd174fa19146cbce97624278, ea42eb67bb0a4b99858a55d8c47da207; no voters:
I20260620 02:59:19.423828 4993 raft_consensus.cc:2781] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:59:19.423967 4993 raft_consensus.cc:482] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:59:19.424062 4993 raft_consensus.cc:3037] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:19.426271 4993 raft_consensus.cc:504] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab1de495f4d4c66b79b779ce6d8b26e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41029 } } peers { permanent_uuid: "ea42eb67bb0a4b99858a55d8c47da207" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43537 } } peers { permanent_uuid: "ea29ed00dd174fa19146cbce97624278" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46871 } }
I20260620 02:59:19.426841 4993 leader_election.cc:288] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [CANDIDATE]: Term 1 election: Requested vote from peers 2ab1de495f4d4c66b79b779ce6d8b26e (127.0.97.1:41029), ea29ed00dd174fa19146cbce97624278 (127.0.97.3:46871)
I20260620 02:59:19.427273 4657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d603c3e0e734c77b397eb00adef0c1b" candidate_uuid: "ea42eb67bb0a4b99858a55d8c47da207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab1de495f4d4c66b79b779ce6d8b26e"
I20260620 02:59:19.427299 4922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d603c3e0e734c77b397eb00adef0c1b" candidate_uuid: "ea42eb67bb0a4b99858a55d8c47da207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea29ed00dd174fa19146cbce97624278"
I20260620 02:59:19.427469 4657 raft_consensus.cc:3037] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:19.427477 4922 raft_consensus.cc:3037] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:19.429523 4657 raft_consensus.cc:2454] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea42eb67bb0a4b99858a55d8c47da207 in term 1.
I20260620 02:59:19.429544 4922 raft_consensus.cc:2454] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea42eb67bb0a4b99858a55d8c47da207 in term 1.
I20260620 02:59:19.429950 4723 leader_election.cc:302] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ab1de495f4d4c66b79b779ce6d8b26e, ea42eb67bb0a4b99858a55d8c47da207; no voters:
I20260620 02:59:19.430227 4993 raft_consensus.cc:2781] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:19.430650 4993 raft_consensus.cc:686] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [term 1 LEADER]: Becoming Leader. State: Replica: ea42eb67bb0a4b99858a55d8c47da207, State: Running, Role: LEADER
I20260620 02:59:19.431169 4993 consensus_queue.cc:237] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab1de495f4d4c66b79b779ce6d8b26e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41029 } } peers { permanent_uuid: "ea42eb67bb0a4b99858a55d8c47da207" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43537 } } peers { permanent_uuid: "ea29ed00dd174fa19146cbce97624278" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46871 } }
I20260620 02:59:19.435726 4514 catalog_manager.cc:5495] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea42eb67bb0a4b99858a55d8c47da207 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "ea42eb67bb0a4b99858a55d8c47da207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab1de495f4d4c66b79b779ce6d8b26e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea42eb67bb0a4b99858a55d8c47da207" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea29ed00dd174fa19146cbce97624278" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46871 } health_report { overall_health: UNKNOWN } } }
I20260620 02:59:19.496172 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:19.491096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.0.97.2:43537)
0620 02:59:19.491275 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:19.491279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:19.491296 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:59:19.491379 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:19.491405 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:19.491514 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:19.491638 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:59:19.491952 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.492706 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.494130 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.494526 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.494615 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:19.495050 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:59:19.495572 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:59:19.495676 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:19.495807 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 02:59:19.495866 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 02:59:19.504338 4657 raft_consensus.cc:1261] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e [term 1 FOLLOWER]: Refusing update from remote peer ea42eb67bb0a4b99858a55d8c47da207: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:19.504341 4922 raft_consensus.cc:1261] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278 [term 1 FOLLOWER]: Refusing update from remote peer ea42eb67bb0a4b99858a55d8c47da207: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:19.505389 4996 consensus_queue.cc:1035] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea29ed00dd174fa19146cbce97624278" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:19.505788 4993 consensus_queue.cc:1035] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab1de495f4d4c66b79b779ce6d8b26e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:19.516255 4999 mvcc.cc:205] Tried to move back new op lower bound from 7298762176519778304 to 7298762176233824256. Current Snapshot: MvccSnapshot[applied={T|T < 7298762176519778304}]
I20260620 02:59:19.516992 5000 mvcc.cc:205] Tried to move back new op lower bound from 7298762176519778304 to 7298762176233824256. Current Snapshot: MvccSnapshot[applied={T|T < 7298762176519778304}]
I20260620 02:59:19.544754 4514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:19.545006 4514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:19.548318 4514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c29c49b95f34447fafb35d2182667cd5: Sending DeleteTablet for 3 replicas of tablet 5d603c3e0e734c77b397eb00adef0c1b
I20260620 02:59:19.549364 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d603c3e0e734c77b397eb00adef0c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38832
I20260620 02:59:19.549700 4770 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d603c3e0e734c77b397eb00adef0c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42052
I20260620 02:59:19.550297 5009 tablet_replica.cc:330] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e: stopping tablet replica
I20260620 02:59:19.550731 5009 raft_consensus.cc:2229] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:19.550830 4902 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d603c3e0e734c77b397eb00adef0c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51894
I20260620 02:59:19.551185 5009 raft_consensus.cc:2258] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:19.553161 5009 ts_tablet_manager.cc:1905] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:19.553725 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 4575
I20260620 02:59:19.560292 5009 ts_tablet_manager.cc:1918] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:59:19.560470 5009 log.cc:1196] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/5d603c3e0e734c77b397eb00adef0c1b
I20260620 02:59:19.561041 5009 ts_tablet_manager.cc:1939] T 5d603c3e0e734c77b397eb00adef0c1b P 2ab1de495f4d4c66b79b779ce6d8b26e: Deleting consensus metadata
I20260620 02:59:19.562914 4498 catalog_manager.cc:4841] TS 2ab1de495f4d4c66b79b779ce6d8b26e (127.0.97.1:41029): tablet 5d603c3e0e734c77b397eb00adef0c1b (table test-table [id=183153c9a1084e249ffd431710a5c7ca]) successfully deleted
I20260620 02:59:19.570135 5011 tablet_replica.cc:330] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278: stopping tablet replica
I20260620 02:59:19.570595 5011 raft_consensus.cc:2229] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:19.571164 5011 raft_consensus.cc:2258] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:19.573309 5011 ts_tablet_manager.cc:1905] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:19.580210 5011 ts_tablet_manager.cc:1918] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:59:19.580377 5011 log.cc:1196] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/5d603c3e0e734c77b397eb00adef0c1b
I20260620 02:59:19.580899 5011 ts_tablet_manager.cc:1939] T 5d603c3e0e734c77b397eb00adef0c1b P ea29ed00dd174fa19146cbce97624278: Deleting consensus metadata
I20260620 02:59:19.582020 5010 tablet_replica.cc:330] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207: stopping tablet replica
I20260620 02:59:19.582551 5010 raft_consensus.cc:2229] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:59:19.582789 4499 catalog_manager.cc:4841] TS ea29ed00dd174fa19146cbce97624278 (127.0.97.3:46871): tablet 5d603c3e0e734c77b397eb00adef0c1b (table test-table [id=183153c9a1084e249ffd431710a5c7ca]) successfully deleted
I20260620 02:59:19.583312 5010 raft_consensus.cc:2258] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 02:59:19 dist-test-slave-f1j0 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:59:19.585392 5010 ts_tablet_manager.cc:1905] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:19.591854 5010 ts_tablet_manager.cc:1918] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:59:19.592029 5010 log.cc:1196] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/5d603c3e0e734c77b397eb00adef0c1b
I20260620 02:59:19.592583 5010 ts_tablet_manager.cc:1939] T 5d603c3e0e734c77b397eb00adef0c1b P ea42eb67bb0a4b99858a55d8c47da207: Deleting consensus metadata
I20260620 02:59:19.592617 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:19.555590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38842 (local address 127.0.97.1:41029)
0620 02:59:19.555809 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:19.555818 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:19.570104 (+ 14286us) server_negotiation.cc:407] Connection header received
0620 02:59:19.570351 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:19.570360 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:19.570470 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:19.570571 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:19.572813 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.574356 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.576811 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.577789 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.577937 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:19.584888 (+ 6951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:19.584905 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:19.584925 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:19.585012 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:19.588970 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:19.589972 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:19.589982 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:19.589988 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:19.590152 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:19.590666 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:19.590672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:19.590675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:19.591119 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:19.591492 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:19.591765 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 02:59:19.592153 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:59:19.593456 4697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:19.594565 4501 catalog_manager.cc:4841] TS ea42eb67bb0a4b99858a55d8c47da207 (127.0.97.2:43537): tablet 5d603c3e0e734c77b397eb00adef0c1b (table test-table [id=183153c9a1084e249ffd431710a5c7ca]) successfully deleted
I20260620 02:59:19.751081 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 4575
I20260620 02:59:19.769503 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 4708
Jun 20 02:59:19 dist-test-slave-f1j0 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:59:19.789261 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:19.770336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.0.97.2:43537)
0620 02:59:19.770502 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:19.770506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:19.770538 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:59:19.770743 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:19.770747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:19.770829 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:19.770906 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:19.772407 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.773210 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.774994 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.775405 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.775497 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:19.783314 (+ 7817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:19.783325 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:19.783340 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:19.783372 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:19.786550 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:19.787259 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:19.787266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:19.787268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:19.787426 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:19.787788 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:19.787791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:19.787793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:19.788150 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:19.788366 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:19.788729 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 02:59:19.788880 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:59:19.790056 4830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:19.902942 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 4708
I20260620 02:59:19.921713 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 4840
Jun 20 02:59:19 dist-test-slave-f1j0 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:59:19.936522 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:19.922491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51906 (local address 127.0.97.3:46871)
0620 02:59:19.922991 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:19.922996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:19.923011 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:59:19.923066 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:19.923069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:19.923131 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:19.923257 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:19.924600 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.925299 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.926633 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:19.927003 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:19.927132 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:19.931237 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:19.931251 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:19.931268 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:19.931305 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:19.933834 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:19.934507 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:19.934514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:19.934517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:19.934671 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:19.935054 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:19.935061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:19.935066 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:19.935390 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:19.935704 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:19.935969 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 02:59:19.936190 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":250,"threads_started":1}
I20260620 02:59:19.937224 4962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:20.025374 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 4840
I20260620 02:59:20.042098 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 4483
I20260620 02:59:20.043313 4544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:20.154198 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 4483
2026-06-20T02:59:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4317 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T02:59:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:59:20Z Disabled control of system clock
I20260620 02:59:20.219864 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41435
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 02:59:20.537621 5021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:20.537905 5021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:20.537989 5021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:20.545097 5021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:59:20.545207 5021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:20.545246 5021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:20.545303 5021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:59:20.545347 5021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:59:20.557623 5021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41435
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:20.558867 5021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:20.560560 5021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:20.571856 5027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:20.571926 5026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:20.571962 5029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:20.572897 5021 server_base.cc:1034] running on GCE node
I20260620 02:59:20.573716 5021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:20.575039 5021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:20.576258 5021 hybrid_clock.cc:648] HybridClock initialized: now 1781924360576224 us; error 48 us; skew 500 ppm
I20260620 02:59:20.576644 5021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:59:20.578905 5021 webserver.cc:460] Webserver started at http://127.0.97.62:42853/ using document root <none> and password file <none>
I20260620 02:59:20.579573 5021 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:20.579654 5021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:20.580003 5021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:20.582808 5021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "76d167de70c64ddb8d9e906df54894f5"
format_stamp: "Formatted at 2026-06-20 02:59:20 on dist-test-slave-f1j0"
server_key: "e494a37a9ef85d6584c551e96c9d472d"
server_key_iv: "0da69cb1c7acaa6a2d08ceab2f9ceb95"
server_key_version: "encryptionkey@0"
I20260620 02:59:20.583572 5021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "76d167de70c64ddb8d9e906df54894f5"
format_stamp: "Formatted at 2026-06-20 02:59:20 on dist-test-slave-f1j0"
server_key: "e494a37a9ef85d6584c551e96c9d472d"
server_key_iv: "0da69cb1c7acaa6a2d08ceab2f9ceb95"
server_key_version: "encryptionkey@0"
I20260620 02:59:20.588781 5021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 02:59:20.592303 5035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:20.593672 5021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:59:20.593845 5021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "76d167de70c64ddb8d9e906df54894f5"
format_stamp: "Formatted at 2026-06-20 02:59:20 on dist-test-slave-f1j0"
server_key: "e494a37a9ef85d6584c551e96c9d472d"
server_key_iv: "0da69cb1c7acaa6a2d08ceab2f9ceb95"
server_key_version: "encryptionkey@0"
I20260620 02:59:20.594059 5021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:20.599769 5021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:20.600497 5021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:20.600786 5021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:20.619374 5021 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41435
I20260620 02:59:20.619406 5086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41435 every 8 connection(s)
I20260620 02:59:20.620726 5021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:59:20.624894 5087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:20.627525 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 5021
I20260620 02:59:20.627750 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 02:59:20.628371 388 external_mini_cluster.cc:1428] Setting key cebe8950b4d2774faeef7bc346b76d07
I20260620 02:59:20.638710 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:20.630507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.0.97.62:41435)
0620 02:59:20.631665 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:20.631680 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:20.631740 (+ 60us) server_negotiation.cc:407] Connection header received
0620 02:59:20.634270 (+ 2530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:20.634313 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:20.634917 (+ 604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:20.635430 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:20.636572 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:20.636582 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:20.636600 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:59:20.636618 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:20.636698 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:59:20.636721 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:20.636960 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:20.637110 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 02:59:20.637381 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":342,"threads_started":1}
I20260620 02:59:20.639333 5087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5: Bootstrap starting.
I20260620 02:59:20.642838 5087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:20.644068 5087 log.cc:824] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:20.646951 5087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5: No bootstrap required, opened a new log
I20260620 02:59:20.654160 5087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d167de70c64ddb8d9e906df54894f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41435 } }
I20260620 02:59:20.654515 5087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:20.654630 5087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d167de70c64ddb8d9e906df54894f5, State: Initialized, Role: FOLLOWER
I20260620 02:59:20.655247 5087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d167de70c64ddb8d9e906df54894f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41435 } }
I20260620 02:59:20.655479 5087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:59:20.655603 5087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:59:20.655769 5087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:20.658154 5087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d167de70c64ddb8d9e906df54894f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41435 } }
I20260620 02:59:20.658640 5087 leader_election.cc:302] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76d167de70c64ddb8d9e906df54894f5; no voters:
I20260620 02:59:20.659209 5087 leader_election.cc:288] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:59:20.659351 5092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:20.660101 5092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5 [term 1 LEADER]: Becoming Leader. State: Replica: 76d167de70c64ddb8d9e906df54894f5, State: Running, Role: LEADER
I20260620 02:59:20.660683 5092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d167de70c64ddb8d9e906df54894f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41435 } }
I20260620 02:59:20.661267 5087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:59:20.663625 5094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76d167de70c64ddb8d9e906df54894f5. Latest consensus state: current_term: 1 leader_uuid: "76d167de70c64ddb8d9e906df54894f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d167de70c64ddb8d9e906df54894f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41435 } } }
I20260620 02:59:20.664047 5094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:20.664113 5093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76d167de70c64ddb8d9e906df54894f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d167de70c64ddb8d9e906df54894f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41435 } } }
I20260620 02:59:20.665226 5093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:20.666548 5100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:59:20.671738 5100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:59:20.680936 5100 catalog_manager.cc:1347] Generated new cluster ID: 4b55b9aa9d6e447387b0fd6dd105dffc
I20260620 02:59:20.681048 5100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:59:20.700002 5100 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:59:20.700433 5100 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:59:20.710417 5100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5: Generated new TSK 0
I20260620 02:59:20.711459 5100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:59:20.727272 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41435
--builtin_ntp_servers=127.0.97.20:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 02:59:21.064675 5111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:21.064955 5111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:21.065042 5111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:21.072522 5111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:21.072616 5111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:21.072722 5111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:59:21.083112 5111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:21.084378 5111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:21.086239 5111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:21.101534 5116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:21.101511 5119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:21.101464 5117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:21.102268 5111 server_base.cc:1034] running on GCE node
I20260620 02:59:21.102872 5111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:21.103904 5111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:21.105126 5111 hybrid_clock.cc:648] HybridClock initialized: now 1781924361105062 us; error 59 us; skew 500 ppm
I20260620 02:59:21.105412 5111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:59:21.107721 5111 webserver.cc:460] Webserver started at http://127.0.97.1:36219/ using document root <none> and password file <none>
I20260620 02:59:21.108392 5111 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:21.108474 5111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:21.108727 5111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:21.111644 5111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "263c9fa1444b47aab4ec1a4572a9e042"
format_stamp: "Formatted at 2026-06-20 02:59:21 on dist-test-slave-f1j0"
server_key: "764eb427a7472d0423de5c1159fbf066"
server_key_iv: "ac991da50f319b03e73820e7b1fb6897"
server_key_version: "encryptionkey@0"
I20260620 02:59:21.112499 5111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "263c9fa1444b47aab4ec1a4572a9e042"
format_stamp: "Formatted at 2026-06-20 02:59:21 on dist-test-slave-f1j0"
server_key: "764eb427a7472d0423de5c1159fbf066"
server_key_iv: "ac991da50f319b03e73820e7b1fb6897"
server_key_version: "encryptionkey@0"
I20260620 02:59:21.119946 5111 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260620 02:59:21.124879 5125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:21.130235 5111 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.002s
I20260620 02:59:21.130511 5111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "263c9fa1444b47aab4ec1a4572a9e042"
format_stamp: "Formatted at 2026-06-20 02:59:21 on dist-test-slave-f1j0"
server_key: "764eb427a7472d0423de5c1159fbf066"
server_key_iv: "ac991da50f319b03e73820e7b1fb6897"
server_key_version: "encryptionkey@0"
I20260620 02:59:21.130833 5111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:21.139147 5111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:21.140236 5111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:21.140532 5111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:21.141667 5111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:21.143039 5111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:21.143122 5111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:21.143205 5111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:21.143260 5111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:21.177910 5111 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38545
I20260620 02:59:21.177963 5237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38545 every 8 connection(s)
I20260620 02:59:21.179239 5111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:59:21.180989 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 5111
I20260620 02:59:21.181187 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:59:21.181896 388 external_mini_cluster.cc:1428] Setting key 5c649e0d8d6d072e09f4763b73d1da4c
I20260620 02:59:21.184914 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41435
--builtin_ntp_servers=127.0.97.20:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 02:59:21.188905 5240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:21.181864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39587 (local address 127.0.97.62:41435)
0620 02:59:21.182711 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:21.182715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:21.183934 (+ 1219us) server_negotiation.cc:407] Connection header received
0620 02:59:21.186553 (+ 2619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:21.186557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:21.186615 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:21.186709 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:21.188183 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:21.188187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:21.188190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:59:21.188205 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:21.188244 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:59:21.188255 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:21.188401 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:21.188634 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 02:59:21.188680 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":610,"threads_started":1}
I20260620 02:59:21.190169 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:21.182679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41435 (local address 127.0.97.1:39587)
0620 02:59:21.183776 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:21.183823 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:21.186273 (+ 2450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:21.186969 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:21.186983 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:21.187103 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:21.187812 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:59:21.187818 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:21.187877 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:59:21.187990 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:21.188403 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:21.188416 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:21.188428 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 02:59:21.188694 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 02:59:21.188946 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":301,"threads_started":1}
I20260620 02:59:21.191628 5238 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41435
I20260620 02:59:21.191910 5238 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:21.192447 5238 heartbeater.cc:511] Master 127.0.97.62:41435 requested a full tablet report, sending...
I20260620 02:59:21.194439 5052 ts_manager.cc:194] Registered new tserver with Master: 263c9fa1444b47aab4ec1a4572a9e042 (127.0.97.1:38545)
W20260620 02:59:21.505180 5242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:21.505513 5242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:21.505687 5242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:21.512951 5242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:21.513092 5242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:21.513273 5242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:59:21.522445 5242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:21.523799 5242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:21.525681 5242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:21.539007 5247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:21.538998 5248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:21.539290 5250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:21.539742 5242 server_base.cc:1034] running on GCE node
I20260620 02:59:21.540433 5242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:21.541376 5242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:21.542623 5242 hybrid_clock.cc:648] HybridClock initialized: now 1781924361542575 us; error 53 us; skew 500 ppm
I20260620 02:59:21.542968 5242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:59:21.545270 5242 webserver.cc:460] Webserver started at http://127.0.97.2:39479/ using document root <none> and password file <none>
I20260620 02:59:21.546056 5242 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:21.546191 5242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:21.546506 5242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:21.549407 5242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "2af15bcfd1a044e08aaa84fec3cbff9a"
format_stamp: "Formatted at 2026-06-20 02:59:21 on dist-test-slave-f1j0"
server_key: "7571f4558b028a32dc626b922122fc56"
server_key_iv: "4375a7004cf6e5e12a0c10dc92fc6114"
server_key_version: "encryptionkey@0"
I20260620 02:59:21.550292 5242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2af15bcfd1a044e08aaa84fec3cbff9a"
format_stamp: "Formatted at 2026-06-20 02:59:21 on dist-test-slave-f1j0"
server_key: "7571f4558b028a32dc626b922122fc56"
server_key_iv: "4375a7004cf6e5e12a0c10dc92fc6114"
server_key_version: "encryptionkey@0"
I20260620 02:59:21.555640 5242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 02:59:21.559157 5256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:21.560570 5242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:59:21.560770 5242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "2af15bcfd1a044e08aaa84fec3cbff9a"
format_stamp: "Formatted at 2026-06-20 02:59:21 on dist-test-slave-f1j0"
server_key: "7571f4558b028a32dc626b922122fc56"
server_key_iv: "4375a7004cf6e5e12a0c10dc92fc6114"
server_key_version: "encryptionkey@0"
I20260620 02:59:21.561034 5242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:21.566756 5242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:21.567648 5242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:21.567909 5242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:21.568804 5242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:21.570194 5242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:21.570317 5242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:21.570438 5242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:21.570524 5242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:21.603003 5242 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39595
I20260620 02:59:21.603048 5368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39595 every 8 connection(s)
I20260620 02:59:21.604363 5242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:59:21.605827 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 5242
I20260620 02:59:21.606011 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:59:21.606565 388 external_mini_cluster.cc:1428] Setting key 5f5bde7fa128a018f64841b80b08d67c
I20260620 02:59:21.609493 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41435
--builtin_ntp_servers=127.0.97.20:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 02:59:21.613446 5240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:21.607349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53027 (local address 127.0.97.62:41435)
0620 02:59:21.607547 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:21.607553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:21.609271 (+ 1718us) server_negotiation.cc:407] Connection header received
0620 02:59:21.611061 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:21.611070 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:21.611136 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:21.611245 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:21.612643 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:21.612653 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:21.612662 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:59:21.612685 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:21.612737 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:59:21.612756 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:21.612879 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:21.613049 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 02:59:21.613116 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:59:21.614784 5371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:21.608016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41435 (local address 127.0.97.2:53027)
0620 02:59:21.609096 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:21.609152 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:21.610791 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:21.611409 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:21.611417 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:21.611511 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:21.612326 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:59:21.612329 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:21.612368 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:59:21.612456 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:21.612854 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:21.612861 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:21.612867 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 02:59:21.613086 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 02:59:21.613337 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":324,"threads_started":1}
I20260620 02:59:21.616238 5369 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41435
I20260620 02:59:21.616550 5369 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:21.617208 5369 heartbeater.cc:511] Master 127.0.97.62:41435 requested a full tablet report, sending...
I20260620 02:59:21.618844 5052 ts_manager.cc:194] Registered new tserver with Master: 2af15bcfd1a044e08aaa84fec3cbff9a (127.0.97.2:39595)
W20260620 02:59:21.929781 5372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:21.930092 5372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:21.930260 5372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:21.937616 5372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:21.937753 5372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:21.937930 5372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:59:21.947049 5372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:21.948442 5372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:21.950263 5372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:21.963047 5377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:21.963083 5378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:21.963572 5380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:21.964217 5372 server_base.cc:1034] running on GCE node
I20260620 02:59:21.964825 5372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:21.965844 5372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:21.967033 5372 hybrid_clock.cc:648] HybridClock initialized: now 1781924361966983 us; error 40 us; skew 500 ppm
I20260620 02:59:21.967402 5372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:59:21.969745 5372 webserver.cc:460] Webserver started at http://127.0.97.3:36119/ using document root <none> and password file <none>
I20260620 02:59:21.970412 5372 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:21.970521 5372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:21.970841 5372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:21.973728 5372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "fd4d9725142248a2b0d838e2bc2301f4"
format_stamp: "Formatted at 2026-06-20 02:59:21 on dist-test-slave-f1j0"
server_key: "a0c965559af9b5d27942deb0d5346186"
server_key_iv: "54088c7e35ba3ecd3d1e2569ba610779"
server_key_version: "encryptionkey@0"
I20260620 02:59:21.974539 5372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fd4d9725142248a2b0d838e2bc2301f4"
format_stamp: "Formatted at 2026-06-20 02:59:21 on dist-test-slave-f1j0"
server_key: "a0c965559af9b5d27942deb0d5346186"
server_key_iv: "54088c7e35ba3ecd3d1e2569ba610779"
server_key_version: "encryptionkey@0"
I20260620 02:59:21.979432 5372 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:59:21.982697 5386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:21.983997 5372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:59:21.984172 5372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "fd4d9725142248a2b0d838e2bc2301f4"
format_stamp: "Formatted at 2026-06-20 02:59:21 on dist-test-slave-f1j0"
server_key: "a0c965559af9b5d27942deb0d5346186"
server_key_iv: "54088c7e35ba3ecd3d1e2569ba610779"
server_key_version: "encryptionkey@0"
I20260620 02:59:21.984444 5372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:21.990332 5372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:21.991072 5372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:21.991348 5372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:21.992187 5372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:21.993623 5372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:21.993752 5372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:21.993891 5372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:21.993974 5372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:22.027376 5372 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42705
I20260620 02:59:22.027427 5498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42705 every 8 connection(s)
I20260620 02:59:22.028740 5372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:59:22.030215 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 5372
I20260620 02:59:22.030417 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:59:22.030965 388 external_mini_cluster.cc:1428] Setting key 8ae34f7fb0d39ff85368f49aff1e4bac
I20260620 02:59:22.038398 5240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.031555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42703 (local address 127.0.97.62:41435)
0620 02:59:22.031760 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:22.031765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:22.033429 (+ 1664us) server_negotiation.cc:407] Connection header received
0620 02:59:22.035554 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:22.035560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:22.035656 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:22.035785 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:22.037450 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:22.037458 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:22.037465 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:59:22.037486 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:22.037521 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:59:22.037555 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:22.037721 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:22.037963 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 02:59:22.038021 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:59:22.039623 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.032147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41435 (local address 127.0.97.3:42703)
0620 02:59:22.033268 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:22.033316 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:22.035231 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:22.036057 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:22.036073 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:22.036216 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:22.037083 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:59:22.037087 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:22.037142 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:59:22.037266 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:22.037677 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:22.037690 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:22.037700 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 02:59:22.038032 (+ 332us) client_negotiation.cc:240] Negotiation successful
0620 02:59:22.038317 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":311,"threads_started":1}
I20260620 02:59:22.041173 5499 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41435
I20260620 02:59:22.041539 5499 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:22.042280 5499 heartbeater.cc:511] Master 127.0.97.62:41435 requested a full tablet report, sending...
I20260620 02:59:22.043859 5052 ts_manager.cc:194] Registered new tserver with Master: fd4d9725142248a2b0d838e2bc2301f4 (127.0.97.3:42705)
I20260620 02:59:22.047531 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:59:22.054648 5240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.051859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60970 (local address 127.0.97.62:41435)
0620 02:59:22.052006 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:22.052010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:22.052543 (+ 533us) server_negotiation.cc:407] Connection header received
0620 02:59:22.052744 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:22.052756 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:22.052804 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:22.052901 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:22.053999 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:22.054006 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:22.054012 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:59:22.054034 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:22.054073 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:59:22.054090 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:22.054224 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:22.054287 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 02:59:22.054343 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 02:59:22.058951 5052 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:59:22.071308 5052 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:59:22.096220 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.085654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42705 (local address 127.0.0.1:59948)
0620 02:59:22.086617 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:22.086658 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:22.086785 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:22.092959 (+ 6174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:22.092967 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:22.093071 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:22.094001 (+ 930us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:59:22.094004 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:22.094039 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:59:22.094121 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:22.095567 (+ 1446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:22.095579 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:22.095586 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 02:59:22.095738 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 02:59:22.095822 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"threads_started":1}
I20260620 02:59:22.097188 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.091701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59948 (local address 127.0.97.3:42705)
0620 02:59:22.092364 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:22.092373 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:22.092408 (+ 35us) server_negotiation.cc:407] Connection header received
0620 02:59:22.092476 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:22.092483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:22.092744 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:22.093551 (+ 807us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:22.095032 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:22.095039 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:22.095053 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:59:22.095067 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:22.095427 (+ 360us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:59:22.095456 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:22.096675 (+ 1219us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:22.096743 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 02:59:22.096833 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":310,"threads_started":1}
I20260620 02:59:22.100016 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.085916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39595 (local address 127.0.0.1:41810)
0620 02:59:22.096410 (+ 10494us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:22.096427 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:22.096496 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:22.098257 (+ 1761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:22.098261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:22.098284 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:22.098872 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:59:22.098875 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:22.098891 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:59:22.098930 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:22.099627 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:22.099632 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:22.099637 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 02:59:22.099717 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 02:59:22.099763 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10406,"thread_start_us":398,"threads_started":1}
I20260620 02:59:22.100821 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.086219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41810 (local address 127.0.97.2:39595)
0620 02:59:22.097172 (+ 10953us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:22.097184 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:22.097224 (+ 40us) server_negotiation.cc:407] Connection header received
0620 02:59:22.097734 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:22.097746 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:22.098034 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:22.098227 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:22.099368 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:22.099380 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:22.099399 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:59:22.099420 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:22.099485 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:59:22.099515 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:22.099734 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:22.099804 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 02:59:22.099921 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10726,"thread_start_us":11192,"threads_started":1}
I20260620 02:59:22.105535 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.101653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38545 (local address 127.0.0.1:38256)
0620 02:59:22.102153 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:22.102170 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:22.102272 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:22.103793 (+ 1521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:22.103797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:22.103819 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:22.104463 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:59:22.104467 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:22.104483 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:59:22.104527 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:22.104976 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:22.104984 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:22.104992 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 02:59:22.105119 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 02:59:22.105174 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367}
I20260620 02:59:22.106331 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.101049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.0.97.1:38545)
0620 02:59:22.101615 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:22.101624 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:22.102252 (+ 628us) server_negotiation.cc:407] Connection header received
0620 02:59:22.102377 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:22.102385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:22.102613 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:22.103775 (+ 1162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:22.104700 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:22.104709 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:22.104722 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:59:22.104740 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:22.104794 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:59:22.104812 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:22.104924 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:22.105717 (+ 793us) server_negotiation.cc:299] Negotiation successful
0620 02:59:22.105810 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":280,"threads_started":1}
I20260620 02:59:22.105062 5304 tablet_service.cc:1467] Processing CreateTablet for tablet a4bf6d2b82724971a8f9e9abb7710c98 (DEFAULT_TABLE table=test-table [id=d100ce940c194129ae8fc90f55ced86e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:22.107520 5304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4bf6d2b82724971a8f9e9abb7710c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:22.108373 5434 tablet_service.cc:1467] Processing CreateTablet for tablet a4bf6d2b82724971a8f9e9abb7710c98 (DEFAULT_TABLE table=test-table [id=d100ce940c194129ae8fc90f55ced86e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:22.109869 5434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4bf6d2b82724971a8f9e9abb7710c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:22.109823 5173 tablet_service.cc:1467] Processing CreateTablet for tablet a4bf6d2b82724971a8f9e9abb7710c98 (DEFAULT_TABLE table=test-table [id=d100ce940c194129ae8fc90f55ced86e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:22.111136 5173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4bf6d2b82724971a8f9e9abb7710c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:22.127995 5518 tablet_bootstrap.cc:492] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a: Bootstrap starting.
I20260620 02:59:22.130316 5517 tablet_bootstrap.cc:492] T a4bf6d2b82724971a8f9e9abb7710c98 P 263c9fa1444b47aab4ec1a4572a9e042: Bootstrap starting.
I20260620 02:59:22.131991 5518 tablet_bootstrap.cc:654] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:22.133314 5518 log.cc:824] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:22.134295 5517 tablet_bootstrap.cc:654] T a4bf6d2b82724971a8f9e9abb7710c98 P 263c9fa1444b47aab4ec1a4572a9e042: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:22.135272 5516 tablet_bootstrap.cc:492] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4: Bootstrap starting.
I20260620 02:59:22.135726 5517 log.cc:824] T a4bf6d2b82724971a8f9e9abb7710c98 P 263c9fa1444b47aab4ec1a4572a9e042: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:22.140486 5516 tablet_bootstrap.cc:654] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:22.141822 5516 log.cc:824] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:22.143000 5517 tablet_bootstrap.cc:492] T a4bf6d2b82724971a8f9e9abb7710c98 P 263c9fa1444b47aab4ec1a4572a9e042: No bootstrap required, opened a new log
I20260620 02:59:22.142998 5518 tablet_bootstrap.cc:492] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a: No bootstrap required, opened a new log
I20260620 02:59:22.143273 5518 ts_tablet_manager.cc:1397] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20260620 02:59:22.143293 5517 ts_tablet_manager.cc:1397] T a4bf6d2b82724971a8f9e9abb7710c98 P 263c9fa1444b47aab4ec1a4572a9e042: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260620 02:59:22.145165 5516 tablet_bootstrap.cc:492] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4: No bootstrap required, opened a new log
I20260620 02:59:22.145429 5516 ts_tablet_manager.cc:1397] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 02:59:22.152454 5517 raft_consensus.cc:348] T a4bf6d2b82724971a8f9e9abb7710c98 P 263c9fa1444b47aab4ec1a4572a9e042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd4d9725142248a2b0d838e2bc2301f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42705 } } peers { permanent_uuid: "2af15bcfd1a044e08aaa84fec3cbff9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39595 } } peers { permanent_uuid: "263c9fa1444b47aab4ec1a4572a9e042" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } }
I20260620 02:59:22.152855 5517 raft_consensus.cc:374] T a4bf6d2b82724971a8f9e9abb7710c98 P 263c9fa1444b47aab4ec1a4572a9e042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:22.152953 5517 raft_consensus.cc:729] T a4bf6d2b82724971a8f9e9abb7710c98 P 263c9fa1444b47aab4ec1a4572a9e042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263c9fa1444b47aab4ec1a4572a9e042, State: Initialized, Role: FOLLOWER
I20260620 02:59:22.153662 5517 consensus_queue.cc:260] T a4bf6d2b82724971a8f9e9abb7710c98 P 263c9fa1444b47aab4ec1a4572a9e042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd4d9725142248a2b0d838e2bc2301f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42705 } } peers { permanent_uuid: "2af15bcfd1a044e08aaa84fec3cbff9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39595 } } peers { permanent_uuid: "263c9fa1444b47aab4ec1a4572a9e042" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } }
I20260620 02:59:22.153822 5516 raft_consensus.cc:348] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd4d9725142248a2b0d838e2bc2301f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42705 } } peers { permanent_uuid: "2af15bcfd1a044e08aaa84fec3cbff9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39595 } } peers { permanent_uuid: "263c9fa1444b47aab4ec1a4572a9e042" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } }
I20260620 02:59:22.154248 5516 raft_consensus.cc:374] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:22.154337 5516 raft_consensus.cc:729] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd4d9725142248a2b0d838e2bc2301f4, State: Initialized, Role: FOLLOWER
I20260620 02:59:22.154909 5238 heartbeater.cc:503] Master 127.0.97.62:41435 was elected leader, sending a full tablet report...
I20260620 02:59:22.155059 5516 consensus_queue.cc:260] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd4d9725142248a2b0d838e2bc2301f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42705 } } peers { permanent_uuid: "2af15bcfd1a044e08aaa84fec3cbff9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39595 } } peers { permanent_uuid: "263c9fa1444b47aab4ec1a4572a9e042" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } }
I20260620 02:59:22.154964 5518 raft_consensus.cc:348] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd4d9725142248a2b0d838e2bc2301f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42705 } } peers { permanent_uuid: "2af15bcfd1a044e08aaa84fec3cbff9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39595 } } peers { permanent_uuid: "263c9fa1444b47aab4ec1a4572a9e042" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } }
I20260620 02:59:22.155383 5518 raft_consensus.cc:374] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:22.155476 5518 raft_consensus.cc:729] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2af15bcfd1a044e08aaa84fec3cbff9a, State: Initialized, Role: FOLLOWER
I20260620 02:59:22.155558 5517 ts_tablet_manager.cc:1428] T a4bf6d2b82724971a8f9e9abb7710c98 P 263c9fa1444b47aab4ec1a4572a9e042: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260620 02:59:22.156263 5518 consensus_queue.cc:260] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd4d9725142248a2b0d838e2bc2301f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42705 } } peers { permanent_uuid: "2af15bcfd1a044e08aaa84fec3cbff9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39595 } } peers { permanent_uuid: "263c9fa1444b47aab4ec1a4572a9e042" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } }
I20260620 02:59:22.156495 5499 heartbeater.cc:503] Master 127.0.97.62:41435 was elected leader, sending a full tablet report...
I20260620 02:59:22.157238 5369 heartbeater.cc:503] Master 127.0.97.62:41435 was elected leader, sending a full tablet report...
I20260620 02:59:22.158659 5518 ts_tablet_manager.cc:1428] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260620 02:59:22.160274 5516 ts_tablet_manager.cc:1428] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
W20260620 02:59:22.181376 5239 tablet.cc:2369] T a4bf6d2b82724971a8f9e9abb7710c98 P 263c9fa1444b47aab4ec1a4572a9e042: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:59:22.279913 5524 raft_consensus.cc:482] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 02:59:22.280354 5500 tablet.cc:2369] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:59:22.280244 5524 raft_consensus.cc:504] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd4d9725142248a2b0d838e2bc2301f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42705 } } peers { permanent_uuid: "2af15bcfd1a044e08aaa84fec3cbff9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39595 } } peers { permanent_uuid: "263c9fa1444b47aab4ec1a4572a9e042" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } }
I20260620 02:59:22.281919 5524 leader_election.cc:288] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd4d9725142248a2b0d838e2bc2301f4 (127.0.97.3:42705), 263c9fa1444b47aab4ec1a4572a9e042 (127.0.97.1:38545)
I20260620 02:59:22.285635 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.282436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36481 (local address 127.0.97.3:42705)
0620 02:59:22.282790 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:22.282799 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:22.283495 (+ 696us) server_negotiation.cc:407] Connection header received
0620 02:59:22.283565 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:22.283571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:22.283641 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:22.283741 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:22.284917 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:22.284926 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:22.284929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:59:22.284944 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:22.284971 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:59:22.284982 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:22.285110 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:22.285259 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 02:59:22.285322 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 02:59:22.285790 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.283533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50215 (local address 127.0.97.1:38545)
0620 02:59:22.283741 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:22.283746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:22.283986 (+ 240us) server_negotiation.cc:407] Connection header received
0620 02:59:22.284063 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:22.284069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:22.284129 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:22.284240 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:22.285101 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:22.285107 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:22.285112 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:59:22.285132 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:22.285169 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:59:22.285180 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:22.285305 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:22.285404 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 02:59:22.285466 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 02:59:22.285852 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.282347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42705 (local address 127.0.97.2:36481)
0620 02:59:22.283183 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:22.283199 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:22.283358 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:22.283900 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:22.283904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:22.283923 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:22.284717 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:59:22.284719 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:22.284732 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:59:22.284768 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:22.285111 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:22.285116 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:22.285123 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 02:59:22.285301 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 02:59:22.285348 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":510,"threads_started":1}
I20260620 02:59:22.286003 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.283279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38545 (local address 127.0.97.2:50215)
0620 02:59:22.283873 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:22.283891 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:22.283979 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:22.284264 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:22.284267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:22.284283 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:22.284911 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:59:22.284914 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:22.284926 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:59:22.284958 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:22.285282 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:22.285286 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:22.285291 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 02:59:22.285446 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 02:59:22.285496 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":303,"threads_started":1}
I20260620 02:59:22.287032 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bf6d2b82724971a8f9e9abb7710c98" candidate_uuid: "2af15bcfd1a044e08aaa84fec3cbff9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263c9fa1444b47aab4ec1a4572a9e042" is_pre_election: true
I20260620 02:59:22.287036 5454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bf6d2b82724971a8f9e9abb7710c98" candidate_uuid: "2af15bcfd1a044e08aaa84fec3cbff9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd4d9725142248a2b0d838e2bc2301f4" is_pre_election: true
I20260620 02:59:22.287611 5454 raft_consensus.cc:2454] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2af15bcfd1a044e08aaa84fec3cbff9a in term 0.
I20260620 02:59:22.287611 5193 raft_consensus.cc:2454] T a4bf6d2b82724971a8f9e9abb7710c98 P 263c9fa1444b47aab4ec1a4572a9e042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2af15bcfd1a044e08aaa84fec3cbff9a in term 0.
I20260620 02:59:22.288343 5260 leader_election.cc:302] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2af15bcfd1a044e08aaa84fec3cbff9a, fd4d9725142248a2b0d838e2bc2301f4; no voters:
I20260620 02:59:22.288739 5524 raft_consensus.cc:2781] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:59:22.288882 5524 raft_consensus.cc:482] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:59:22.288988 5524 raft_consensus.cc:3037] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:22.291188 5524 raft_consensus.cc:504] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd4d9725142248a2b0d838e2bc2301f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42705 } } peers { permanent_uuid: "2af15bcfd1a044e08aaa84fec3cbff9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39595 } } peers { permanent_uuid: "263c9fa1444b47aab4ec1a4572a9e042" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } }
I20260620 02:59:22.291821 5524 leader_election.cc:288] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [CANDIDATE]: Term 1 election: Requested vote from peers fd4d9725142248a2b0d838e2bc2301f4 (127.0.97.3:42705), 263c9fa1444b47aab4ec1a4572a9e042 (127.0.97.1:38545)
I20260620 02:59:22.292215 5454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bf6d2b82724971a8f9e9abb7710c98" candidate_uuid: "2af15bcfd1a044e08aaa84fec3cbff9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd4d9725142248a2b0d838e2bc2301f4"
I20260620 02:59:22.292301 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bf6d2b82724971a8f9e9abb7710c98" candidate_uuid: "2af15bcfd1a044e08aaa84fec3cbff9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263c9fa1444b47aab4ec1a4572a9e042"
I20260620 02:59:22.292397 5454 raft_consensus.cc:3037] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:22.292539 5193 raft_consensus.cc:3037] T a4bf6d2b82724971a8f9e9abb7710c98 P 263c9fa1444b47aab4ec1a4572a9e042 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:22.294608 5454 raft_consensus.cc:2454] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2af15bcfd1a044e08aaa84fec3cbff9a in term 1.
I20260620 02:59:22.294746 5193 raft_consensus.cc:2454] T a4bf6d2b82724971a8f9e9abb7710c98 P 263c9fa1444b47aab4ec1a4572a9e042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2af15bcfd1a044e08aaa84fec3cbff9a in term 1.
I20260620 02:59:22.294991 5260 leader_election.cc:302] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2af15bcfd1a044e08aaa84fec3cbff9a, fd4d9725142248a2b0d838e2bc2301f4; no voters:
I20260620 02:59:22.295251 5524 raft_consensus.cc:2781] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:22.295725 5524 raft_consensus.cc:686] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [term 1 LEADER]: Becoming Leader. State: Replica: 2af15bcfd1a044e08aaa84fec3cbff9a, State: Running, Role: LEADER
I20260620 02:59:22.296176 5524 consensus_queue.cc:237] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd4d9725142248a2b0d838e2bc2301f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42705 } } peers { permanent_uuid: "2af15bcfd1a044e08aaa84fec3cbff9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39595 } } peers { permanent_uuid: "263c9fa1444b47aab4ec1a4572a9e042" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } }
I20260620 02:59:22.301188 5051 catalog_manager.cc:5495] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2af15bcfd1a044e08aaa84fec3cbff9a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "2af15bcfd1a044e08aaa84fec3cbff9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd4d9725142248a2b0d838e2bc2301f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2af15bcfd1a044e08aaa84fec3cbff9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "263c9fa1444b47aab4ec1a4572a9e042" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } health_report { overall_health: UNKNOWN } } }
I20260620 02:59:22.331964 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.330010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41822 (local address 127.0.97.2:39595)
0620 02:59:22.330181 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:22.330185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:22.330203 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:59:22.330383 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:22.330387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:22.330472 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:22.330577 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:22.331407 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:22.331411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:22.331414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:59:22.331428 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:22.331455 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:59:22.331465 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:22.331564 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:22.331624 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 02:59:22.331667 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:59:22.339947 5193 raft_consensus.cc:1261] T a4bf6d2b82724971a8f9e9abb7710c98 P 263c9fa1444b47aab4ec1a4572a9e042 [term 1 FOLLOWER]: Refusing update from remote peer 2af15bcfd1a044e08aaa84fec3cbff9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:22.339991 5454 raft_consensus.cc:1261] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4 [term 1 FOLLOWER]: Refusing update from remote peer 2af15bcfd1a044e08aaa84fec3cbff9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:22.340880 5527 consensus_queue.cc:1035] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd4d9725142248a2b0d838e2bc2301f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:22.341281 5524 consensus_queue.cc:1035] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "263c9fa1444b47aab4ec1a4572a9e042" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:22.352350 5532 mvcc.cc:205] Tried to move back new op lower bound from 7298762188134830080 to 7298762187969236992. Current Snapshot: MvccSnapshot[applied={T|T < 7298762188134830080}]
I20260620 02:59:22.354799 5531 mvcc.cc:205] Tried to move back new op lower bound from 7298762188134830080 to 7298762187969236992. Current Snapshot: MvccSnapshot[applied={T|T < 7298762188134830080}]
I20260620 02:59:22.369704 5052 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:22.369977 5052 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:22.373380 5052 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76d167de70c64ddb8d9e906df54894f5: Sending DeleteTablet for 3 replicas of tablet a4bf6d2b82724971a8f9e9abb7710c98
I20260620 02:59:22.374588 5434 tablet_service.cc:1514] Processing DeleteTablet for tablet a4bf6d2b82724971a8f9e9abb7710c98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:22 UTC) from {username='slave'} at 127.0.0.1:59948
I20260620 02:59:22.374573 5304 tablet_service.cc:1514] Processing DeleteTablet for tablet a4bf6d2b82724971a8f9e9abb7710c98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:22 UTC) from {username='slave'} at 127.0.0.1:41810
I20260620 02:59:22.375334 5173 tablet_service.cc:1514] Processing DeleteTablet for tablet a4bf6d2b82724971a8f9e9abb7710c98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:22 UTC) from {username='slave'} at 127.0.0.1:38256
I20260620 02:59:22.375501 5540 tablet_replica.cc:330] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a: stopping tablet replica
I20260620 02:59:22.375954 5540 raft_consensus.cc:2229] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:59:22.376681 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 5111
I20260620 02:59:22.376725 5540 raft_consensus.cc:2258] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:22.377509 5541 tablet_replica.cc:330] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4: stopping tablet replica
I20260620 02:59:22.378034 5541 raft_consensus.cc:2229] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:22.378587 5541 raft_consensus.cc:2258] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:22.382079 5542 tablet_replica.cc:330] T a4bf6d2b82724971a8f9e9abb7710c98 P 263c9fa1444b47aab4ec1a4572a9e042: stopping tablet replica
I20260620 02:59:22.382507 5540 ts_tablet_manager.cc:1905] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:22.382503 5542 raft_consensus.cc:2229] T a4bf6d2b82724971a8f9e9abb7710c98 P 263c9fa1444b47aab4ec1a4572a9e042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:22.382988 5542 raft_consensus.cc:2258] T a4bf6d2b82724971a8f9e9abb7710c98 P 263c9fa1444b47aab4ec1a4572a9e042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:22.383909 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.378160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38272 (local address 127.0.97.1:38545)
0620 02:59:22.378348 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:22.378356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:22.381334 (+ 2978us) server_negotiation.cc:407] Connection header received
0620 02:59:22.381550 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:22.381555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:22.381714 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:22.381832 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:22.383004 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:22.383013 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:22.383019 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:59:22.383041 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:22.383077 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:59:22.383096 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:22.383192 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:22.383472 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 02:59:22.383532 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:59:22.383917 5541 ts_tablet_manager.cc:1905] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:22.385038 5542 ts_tablet_manager.cc:1905] T a4bf6d2b82724971a8f9e9abb7710c98 P 263c9fa1444b47aab4ec1a4572a9e042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:22.385061 5233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:22.391491 5541 ts_tablet_manager.cc:1918] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:59:22.391631 5540 ts_tablet_manager.cc:1918] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:59:22.391747 5541 log.cc:1196] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/a4bf6d2b82724971a8f9e9abb7710c98
I20260620 02:59:22.391999 5540 log.cc:1196] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/a4bf6d2b82724971a8f9e9abb7710c98
I20260620 02:59:22.392318 5541 ts_tablet_manager.cc:1939] T a4bf6d2b82724971a8f9e9abb7710c98 P fd4d9725142248a2b0d838e2bc2301f4: Deleting consensus metadata
I20260620 02:59:22.392747 5540 ts_tablet_manager.cc:1939] T a4bf6d2b82724971a8f9e9abb7710c98 P 2af15bcfd1a044e08aaa84fec3cbff9a: Deleting consensus metadata
I20260620 02:59:22.394409 5039 catalog_manager.cc:4841] TS fd4d9725142248a2b0d838e2bc2301f4 (127.0.97.3:42705): tablet a4bf6d2b82724971a8f9e9abb7710c98 (table test-table [id=d100ce940c194129ae8fc90f55ced86e]) successfully deleted
I20260620 02:59:22.394824 5038 catalog_manager.cc:4841] TS 2af15bcfd1a044e08aaa84fec3cbff9a (127.0.97.2:39595): tablet a4bf6d2b82724971a8f9e9abb7710c98 (table test-table [id=d100ce940c194129ae8fc90f55ced86e]) successfully deleted
I20260620 02:59:22.482076 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 5111
I20260620 02:59:22.499776 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 5242
W20260620 02:59:22.499770 5039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38545 (error 108)
I20260620 02:59:22.501811 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.501268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38545 (local address 127.0.0.1:38276)
0620 02:59:22.501463 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:22.501636 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 02:59:22.502089 5039 catalog_manager.cc:4584] TS 263c9fa1444b47aab4ec1a4572a9e042 (127.0.97.1:38545): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4bf6d2b82724971a8f9e9abb7710c98: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38545: connect: Connection refused (error 111)
I20260620 02:59:22.503257 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.500973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41834 (local address 127.0.97.2:39595)
0620 02:59:22.501184 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:22.501188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:22.501204 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:59:22.501273 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:22.501277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:22.501362 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:22.501462 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:22.502617 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:22.502623 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:22.502627 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:59:22.502652 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:22.502681 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:59:22.502692 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:22.502848 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:22.502921 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 02:59:22.502982 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 02:59:22.503975 5364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:22.555649 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.555138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38545 (local address 127.0.0.1:38278)
0620 02:59:22.555344 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:22.555464 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 02:59:22.556679 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.556320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38545 (local address 127.0.0.1:38280)
0620 02:59:22.556468 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:22.556549 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 02:59:22.612078 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 5242
I20260620 02:59:22.617235 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.616679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38545 (local address 127.0.0.1:38294)
0620 02:59:22.616927 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:22.617036 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 02:59:22.618351 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.617919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38545 (local address 127.0.0.1:38310)
0620 02:59:22.618091 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:22.618186 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 02:59:22.630026 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 5372
I20260620 02:59:22.632819 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.630831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.0.97.3:42705)
0620 02:59:22.631029 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:22.631036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:22.631056 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:59:22.631161 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:22.631166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:22.631255 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:22.631352 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:22.632231 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:22.632238 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:22.632245 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:59:22.632266 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:22.632302 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:59:22.632316 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:22.632437 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:22.632492 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 02:59:22.632542 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:59:22.633664 5494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:22.698964 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.698452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38545 (local address 127.0.0.1:38318)
0620 02:59:22.698702 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:22.698799 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 02:59:22.700100 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:22.699711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38545 (local address 127.0.0.1:38332)
0620 02:59:22.699899 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:22.699977 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 02:59:22.720973 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 5372
I20260620 02:59:22.737346 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 5021
I20260620 02:59:22.738562 5082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:22.842070 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 5021
2026-06-20T02:59:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2686 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 02:59:22.883179 388 mini_oidc.cc:152] Starting JWKS server
I20260620 02:59:22.885851 388 webserver.cc:460] Webserver started at https://127.0.0.1:40721/ using document root <none> and password file <none>
I20260620 02:59:22.886178 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 02:59:22.886827 388 webserver.cc:460] Webserver started at http://127.0.0.1:33525/ using document root <none> and password file <none>
2026-06-20T02:59:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:59:22Z Disabled control of system clock
I20260620 02:59:22.913403 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35381
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35381
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40721/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0/testchainca.cert with env {}
W20260620 02:59:23.229337 5554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:23.229702 5554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:23.229859 5554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:23.237032 5554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:59:23.237207 5554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:23.237304 5554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:23.237393 5554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:59:23.237465 5554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:59:23.237552 5554 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:59:23.237646 5554 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40721/jwks.json
I20260620 02:59:23.247496 5554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35381
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40721/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:23.248893 5554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:23.250259 5554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:23.262333 5559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:23.262403 5562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:23.262439 5560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:23.263262 5554 server_base.cc:1034] running on GCE node
I20260620 02:59:23.264017 5554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:23.265412 5554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:23.266646 5554 hybrid_clock.cc:648] HybridClock initialized: now 1781924363266598 us; error 60 us; skew 500 ppm
I20260620 02:59:23.267005 5554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:59:23.269256 5554 webserver.cc:460] Webserver started at http://127.0.97.62:45605/ using document root <none> and password file <none>
I20260620 02:59:23.270010 5554 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:23.270159 5554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:23.270475 5554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:23.273339 5554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "0bbbb40e944241819660959859dfe207"
format_stamp: "Formatted at 2026-06-20 02:59:23 on dist-test-slave-f1j0"
server_key: "e716653b241e28f2c47a36f1e3dddd7d"
server_key_iv: "d3d13d6cf3b7eb9d5a1877fefd77fda1"
server_key_version: "encryptionkey@0"
I20260620 02:59:23.274192 5554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "0bbbb40e944241819660959859dfe207"
format_stamp: "Formatted at 2026-06-20 02:59:23 on dist-test-slave-f1j0"
server_key: "e716653b241e28f2c47a36f1e3dddd7d"
server_key_iv: "d3d13d6cf3b7eb9d5a1877fefd77fda1"
server_key_version: "encryptionkey@0"
I20260620 02:59:23.279206 5554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:59:23.282618 5568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:23.283883 5554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:59:23.284070 5554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "0bbbb40e944241819660959859dfe207"
format_stamp: "Formatted at 2026-06-20 02:59:23 on dist-test-slave-f1j0"
server_key: "e716653b241e28f2c47a36f1e3dddd7d"
server_key_iv: "d3d13d6cf3b7eb9d5a1877fefd77fda1"
server_key_version: "encryptionkey@0"
I20260620 02:59:23.284298 5554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:23.316927 5554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:23.317852 5554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:23.318156 5554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:23.336349 5554 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35381
I20260620 02:59:23.336369 5619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35381 every 8 connection(s)
I20260620 02:59:23.337700 5554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:59:23.341073 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 5554
I20260620 02:59:23.341264 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 02:59:23.341621 5620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:23.342108 388 external_mini_cluster.cc:1428] Setting key cd3c4f110e3402d8ee501cdbc9f7f757
I20260620 02:59:23.355863 5620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207: Bootstrap starting.
I20260620 02:59:23.356423 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:23.343917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59374 (local address 127.0.97.62:35381)
0620 02:59:23.344949 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:23.344962 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:23.345004 (+ 42us) server_negotiation.cc:407] Connection header received
0620 02:59:23.346541 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:23.346568 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:23.347009 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:23.347339 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:23.348007 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:23.349926 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:23.351799 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:23.352513 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:23.352675 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:23.353994 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:23.354002 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:23.354017 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:59:23.354604 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:23.354682 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:59:23.354703 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:23.354788 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:23.355023 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 02:59:23.355373 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":407,"threads_started":1}
I20260620 02:59:23.359174 5620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:23.360257 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:59:23.360342 5620 log.cc:824] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:23.363427 5620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207: No bootstrap required, opened a new log
I20260620 02:59:23.370908 5620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbbb40e944241819660959859dfe207" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35381 } }
I20260620 02:59:23.371289 5620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:23.371398 5620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bbbb40e944241819660959859dfe207, State: Initialized, Role: FOLLOWER
I20260620 02:59:23.372054 5620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbbb40e944241819660959859dfe207" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35381 } }
I20260620 02:59:23.372284 5620 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:59:23.372416 5620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:59:23.372586 5620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:23.374816 5620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbbb40e944241819660959859dfe207" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35381 } }
I20260620 02:59:23.375355 5620 leader_election.cc:302] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bbbb40e944241819660959859dfe207; no voters:
I20260620 02:59:23.375900 5620 leader_election.cc:288] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:59:23.376065 5625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:23.377036 5625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207 [term 1 LEADER]: Becoming Leader. State: Replica: 0bbbb40e944241819660959859dfe207, State: Running, Role: LEADER
I20260620 02:59:23.377671 5625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbbb40e944241819660959859dfe207" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35381 } }
I20260620 02:59:23.378250 5620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:59:23.380829 5627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bbbb40e944241819660959859dfe207. Latest consensus state: current_term: 1 leader_uuid: "0bbbb40e944241819660959859dfe207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbbb40e944241819660959859dfe207" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35381 } } }
I20260620 02:59:23.381165 5627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:23.380935 5626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bbbb40e944241819660959859dfe207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbbb40e944241819660959859dfe207" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35381 } } }
I20260620 02:59:23.381350 5626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:23.384318 5634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:59:23.389371 5634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:59:23.397817 5634 catalog_manager.cc:1347] Generated new cluster ID: e26c83145d6e4948ae632e2af087dcce
I20260620 02:59:23.397951 5634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:59:23.418700 5634 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:59:23.419955 5634 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:59:23.430526 5634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bbbb40e944241819660959859dfe207: Generated new TSK 0
I20260620 02:59:23.431381 5634 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 02:59:23.473294 5650 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 02:59:23.473491 5650 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 02:59:23.474763 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:23.454122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42962 (local address 127.0.97.62:35381)
0620 02:59:23.454282 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:23.454306 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:23.454800 (+ 494us) server_negotiation.cc:407] Connection header received
0620 02:59:23.455185 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:23.455189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:23.455264 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:23.455414 (+ 150us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:59:23.455957 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:23.456697 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:23.458180 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:23.458628 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:23.458755 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:23.459280 (+ 525us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:59:23.473942 (+ 14662us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40721/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 02:59:23.474432 (+ 490us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42962: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40721/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 02:59:23.474890 5623 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42962: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40721/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 02:59:23.476238 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:23.454072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35381 (local address 127.0.0.1:42962)
0620 02:59:23.454689 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:23.454708 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:23.454912 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:23.455442 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:23.455446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:23.455461 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:59:23.455795 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:23.455804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:23.456837 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:23.456843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:23.458011 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:23.458022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:23.458755 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:23.458824 (+ 69us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:23.458952 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:23.459004 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:59:23.474330 (+ 15326us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40721/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 02:59:23.474688 (+ 358us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:35381: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40721/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":326,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:35381: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40721/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 02:59:23.478205 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 5554
I20260620 02:59:23.479471 5615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:23.572343 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 5554
2026-06-20T02:59:23Z chronyd exiting
I20260620 02:59:23.622769 388 test_util.cc:175] -----------------------------------------------
I20260620 02:59:23.622896 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781924320356866-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (744 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 02:59:23.626709 388 mini_oidc.cc:152] Starting JWKS server
I20260620 02:59:23.628554 388 webserver.cc:460] Webserver started at https://127.0.0.1:42589/ using document root <none> and password file <none>
I20260620 02:59:23.628813 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 02:59:23.629488 388 webserver.cc:460] Webserver started at http://127.0.0.1:41531/ using document root <none> and password file <none>
2026-06-20T02:59:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:59:23Z Disabled control of system clock
I20260620 02:59:23.654291 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43509
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43509
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42589/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924320356866-388-0/testchainca.cert with env {}
W20260620 02:59:23.972494 5657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:23.972821 5657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:23.972980 5657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:23.980137 5657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:59:23.980280 5657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:23.980368 5657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:23.980480 5657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:59:23.980558 5657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:59:23.980645 5657 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:59:23.980741 5657 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42589/jwks.json
I20260620 02:59:23.990648 5657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43509
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42589/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924320356866-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:23.992100 5657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:23.993386 5657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:24.005455 5665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:24.005455 5663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:24.005518 5662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:24.006142 5657 server_base.cc:1034] running on GCE node
I20260620 02:59:24.007004 5657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:24.008402 5657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:24.009646 5657 hybrid_clock.cc:648] HybridClock initialized: now 1781924364009594 us; error 48 us; skew 500 ppm
I20260620 02:59:24.010016 5657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:59:24.012382 5657 webserver.cc:460] Webserver started at http://127.0.97.62:42935/ using document root <none> and password file <none>
I20260620 02:59:24.013077 5657 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:24.013164 5657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:24.013530 5657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:24.016422 5657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "8b9cb1bc9dbb4e20b1cd4a90be611804"
format_stamp: "Formatted at 2026-06-20 02:59:24 on dist-test-slave-f1j0"
server_key: "02ce2400a15ae03703d7d720b7d883b9"
server_key_iv: "cfbab4bef35c6f96c64be75993a05ae5"
server_key_version: "encryptionkey@0"
I20260620 02:59:24.017233 5657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "8b9cb1bc9dbb4e20b1cd4a90be611804"
format_stamp: "Formatted at 2026-06-20 02:59:24 on dist-test-slave-f1j0"
server_key: "02ce2400a15ae03703d7d720b7d883b9"
server_key_iv: "cfbab4bef35c6f96c64be75993a05ae5"
server_key_version: "encryptionkey@0"
I20260620 02:59:24.022528 5657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:59:24.025970 5671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:24.027357 5657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 02:59:24.027540 5657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "8b9cb1bc9dbb4e20b1cd4a90be611804"
format_stamp: "Formatted at 2026-06-20 02:59:24 on dist-test-slave-f1j0"
server_key: "02ce2400a15ae03703d7d720b7d883b9"
server_key_iv: "cfbab4bef35c6f96c64be75993a05ae5"
server_key_version: "encryptionkey@0"
I20260620 02:59:24.027781 5657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:24.039237 5657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:24.040010 5657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:24.040302 5657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:24.058348 5657 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43509
I20260620 02:59:24.058367 5722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43509 every 8 connection(s)
I20260620 02:59:24.059630 5657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:59:24.061513 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 5657
I20260620 02:59:24.061750 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 02:59:24.062515 388 external_mini_cluster.cc:1428] Setting key 28e40e2a8b70ca1d29fdfd0a9df2a993
I20260620 02:59:24.063934 5723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:24.075269 5726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:24.064410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39296 (local address 127.0.97.62:43509)
0620 02:59:24.065308 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:24.065318 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:24.065358 (+ 40us) server_negotiation.cc:407] Connection header received
0620 02:59:24.066889 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:24.066910 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:24.067339 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:24.067735 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:24.068294 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:24.069704 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:24.070995 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:24.071639 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:24.071832 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:24.073004 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:24.073014 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:24.073027 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:59:24.073643 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:24.073721 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:59:24.073746 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:24.073873 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:24.074086 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 02:59:24.074373 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":326,"threads_started":1}
I20260620 02:59:24.075562 5723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804: Bootstrap starting.
I20260620 02:59:24.078814 5723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:24.078838 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:59:24.079730 5723 log.cc:824] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:24.082636 5723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804: No bootstrap required, opened a new log
I20260620 02:59:24.090634 5723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9cb1bc9dbb4e20b1cd4a90be611804" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43509 } }
I20260620 02:59:24.091135 5723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:24.091262 5723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9cb1bc9dbb4e20b1cd4a90be611804, State: Initialized, Role: FOLLOWER
I20260620 02:59:24.091921 5723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9cb1bc9dbb4e20b1cd4a90be611804" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43509 } }
I20260620 02:59:24.092198 5723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:59:24.092339 5723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:59:24.092535 5723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:24.094889 5723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9cb1bc9dbb4e20b1cd4a90be611804" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43509 } }
I20260620 02:59:24.095449 5723 leader_election.cc:302] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b9cb1bc9dbb4e20b1cd4a90be611804; no voters:
I20260620 02:59:24.096022 5723 leader_election.cc:288] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:59:24.096185 5728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:24.096879 5728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804 [term 1 LEADER]: Becoming Leader. State: Replica: 8b9cb1bc9dbb4e20b1cd4a90be611804, State: Running, Role: LEADER
I20260620 02:59:24.097407 5728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9cb1bc9dbb4e20b1cd4a90be611804" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43509 } }
I20260620 02:59:24.098114 5723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:59:24.100292 5729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b9cb1bc9dbb4e20b1cd4a90be611804" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9cb1bc9dbb4e20b1cd4a90be611804" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43509 } } }
I20260620 02:59:24.100769 5729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:24.102535 5737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:59:24.102454 5730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b9cb1bc9dbb4e20b1cd4a90be611804. Latest consensus state: current_term: 1 leader_uuid: "8b9cb1bc9dbb4e20b1cd4a90be611804" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9cb1bc9dbb4e20b1cd4a90be611804" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43509 } } }
I20260620 02:59:24.102908 5730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:24.107863 5737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:59:24.118686 5737 catalog_manager.cc:1347] Generated new cluster ID: 7cc643180dc642369209d6636b493329
I20260620 02:59:24.118877 5737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:59:24.160498 5737 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:59:24.161855 5737 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:59:24.169162 5737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b9cb1bc9dbb4e20b1cd4a90be611804: Generated new TSK 0
I20260620 02:59:24.170032 5737 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 02:59:24.213244 5753 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 02:59:24.213371 5753 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 02:59:24.214490 5726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:24.194303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39312 (local address 127.0.97.62:43509)
0620 02:59:24.194538 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:24.194548 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:24.194998 (+ 450us) server_negotiation.cc:407] Connection header received
0620 02:59:24.195240 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:24.195244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:24.195310 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:24.195434 (+ 124us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:59:24.195968 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:24.196656 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:24.198714 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:24.199495 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:24.199614 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:24.200204 (+ 590us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:59:24.213956 (+ 13752us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42589/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 02:59:24.214259 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39312: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42589/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 02:59:24.214632 5726 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39312: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42589/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 02:59:24.214891 5752 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:24.194252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43509 (local address 127.0.0.1:39312)
0620 02:59:24.194890 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:24.194910 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:24.195071 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:24.195478 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:24.195483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:24.195496 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:59:24.195794 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:24.195807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:24.196787 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:24.196793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:24.198566 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:24.198581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:24.199631 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:24.199680 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:24.199890 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:24.199935 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:59:24.214207 (+ 14272us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42589/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 02:59:24.214488 (+ 281us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:43509: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42589/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":333,"threads_started":1}
I20260620 02:59:24.216336 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 5657
I20260620 02:59:24.217398 5718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:24.312018 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 5657
2026-06-20T02:59:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (740 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 02:59:24.367853 388 mini_oidc.cc:152] Starting JWKS server
I20260620 02:59:24.369895 388 webserver.cc:460] Webserver started at https://127.0.0.1:37883/ using document root <none> and password file <none>
I20260620 02:59:24.370152 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 02:59:24.370787 388 webserver.cc:460] Webserver started at http://127.0.0.1:40013/ using document root <none> and password file <none>
2026-06-20T02:59:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:59:24Z Disabled control of system clock
I20260620 02:59:24.397378 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45029
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45029
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37883/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 02:59:24.715193 5760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:24.715476 5760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:24.715561 5760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:24.722671 5760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:59:24.722784 5760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:24.722820 5760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:24.722869 5760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:59:24.722914 5760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:59:24.722971 5760 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:59:24.723011 5760 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37883/jwks.json
I20260620 02:59:24.732038 5760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37883/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:24.733319 5760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:24.735177 5760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:24.747115 5765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:24.747367 5768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:24.747131 5766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:24.747790 5760 server_base.cc:1034] running on GCE node
I20260620 02:59:24.748701 5760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:24.750149 5760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:24.751410 5760 hybrid_clock.cc:648] HybridClock initialized: now 1781924364751379 us; error 42 us; skew 500 ppm
I20260620 02:59:24.751781 5760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:59:24.754067 5760 webserver.cc:460] Webserver started at http://127.0.97.62:40101/ using document root <none> and password file <none>
I20260620 02:59:24.754786 5760 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:24.754916 5760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:24.755252 5760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:24.758143 5760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "226da5df155149f1bf96977b0f76e2df"
format_stamp: "Formatted at 2026-06-20 02:59:24 on dist-test-slave-f1j0"
server_key: "6c6813f77b7cc75850d187c09e56f0bf"
server_key_iv: "60a48a56c5dba89ee2bdb83b275ba75b"
server_key_version: "encryptionkey@0"
I20260620 02:59:24.758931 5760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "226da5df155149f1bf96977b0f76e2df"
format_stamp: "Formatted at 2026-06-20 02:59:24 on dist-test-slave-f1j0"
server_key: "6c6813f77b7cc75850d187c09e56f0bf"
server_key_iv: "60a48a56c5dba89ee2bdb83b275ba75b"
server_key_version: "encryptionkey@0"
I20260620 02:59:24.763952 5760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:59:24.767381 5774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:24.768772 5760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:59:24.768976 5760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "226da5df155149f1bf96977b0f76e2df"
format_stamp: "Formatted at 2026-06-20 02:59:24 on dist-test-slave-f1j0"
server_key: "6c6813f77b7cc75850d187c09e56f0bf"
server_key_iv: "60a48a56c5dba89ee2bdb83b275ba75b"
server_key_version: "encryptionkey@0"
I20260620 02:59:24.769268 5760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:24.822309 5760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:24.823120 5760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:24.823369 5760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:24.841944 5760 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45029
I20260620 02:59:24.841975 5825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45029 every 8 connection(s)
I20260620 02:59:24.843302 5760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:59:24.846562 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 5760
I20260620 02:59:24.846758 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 02:59:24.847175 5826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:24.847416 388 external_mini_cluster.cc:1428] Setting key 464239dd5156ed727afbadeab47cda95
I20260620 02:59:24.862681 5829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:24.849369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.0.97.62:45029)
0620 02:59:24.850500 (+ 1131us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:24.850515 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:24.850568 (+ 53us) server_negotiation.cc:407] Connection header received
0620 02:59:24.852381 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:24.852412 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:24.853045 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:24.853516 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:24.854096 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:24.856140 (+ 2044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:24.857398 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:24.858481 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:24.858669 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:24.859790 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:24.859801 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:24.859822 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:59:24.860592 (+ 770us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:24.860717 (+ 125us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:59:24.860746 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:24.860863 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:24.861058 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 02:59:24.861382 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":365,"threads_started":1}
I20260620 02:59:24.862968 5826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df: Bootstrap starting.
I20260620 02:59:24.866248 5826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:24.866384 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:59:24.867470 5826 log.cc:824] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:24.870311 5826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df: No bootstrap required, opened a new log
I20260620 02:59:24.877341 5826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226da5df155149f1bf96977b0f76e2df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45029 } }
I20260620 02:59:24.877728 5826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:24.877808 5826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226da5df155149f1bf96977b0f76e2df, State: Initialized, Role: FOLLOWER
I20260620 02:59:24.878336 5826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226da5df155149f1bf96977b0f76e2df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45029 } }
I20260620 02:59:24.878532 5826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:59:24.878640 5826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:59:24.878762 5826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:24.880759 5826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226da5df155149f1bf96977b0f76e2df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45029 } }
I20260620 02:59:24.881246 5826 leader_election.cc:302] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 226da5df155149f1bf96977b0f76e2df; no voters:
I20260620 02:59:24.881829 5826 leader_election.cc:288] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:59:24.881944 5831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:24.882652 5831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df [term 1 LEADER]: Becoming Leader. State: Replica: 226da5df155149f1bf96977b0f76e2df, State: Running, Role: LEADER
I20260620 02:59:24.883140 5831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226da5df155149f1bf96977b0f76e2df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45029 } }
I20260620 02:59:24.883846 5826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:59:24.885918 5833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 226da5df155149f1bf96977b0f76e2df. Latest consensus state: current_term: 1 leader_uuid: "226da5df155149f1bf96977b0f76e2df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226da5df155149f1bf96977b0f76e2df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45029 } } }
I20260620 02:59:24.886379 5833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:24.888638 5832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "226da5df155149f1bf96977b0f76e2df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226da5df155149f1bf96977b0f76e2df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45029 } } }
I20260620 02:59:24.888988 5832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:24.890674 5836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:59:24.897179 5836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:59:24.906023 5836 catalog_manager.cc:1347] Generated new cluster ID: 5285826414504d3fafc351d90a0dd571
I20260620 02:59:24.906198 5836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:59:24.932065 5836 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:59:24.933370 5836 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:59:24.943459 5836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 226da5df155149f1bf96977b0f76e2df: Generated new TSK 0
I20260620 02:59:24.944281 5836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:59:24.971307 5829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:24.962723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50022 (local address 127.0.97.62:45029)
0620 02:59:24.962914 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:24.962920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:24.963446 (+ 526us) server_negotiation.cc:407] Connection header received
0620 02:59:24.963671 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:24.963676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:24.963768 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:24.963922 (+ 154us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:59:24.964479 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:24.965652 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:24.967008 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:24.967696 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:24.967822 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:24.968366 (+ 544us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:59:24.970669 (+ 2303us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37883/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 02:59:24.970991 (+ 322us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50022: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37883/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 02:59:24.971427 5829 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50022: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37883/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 02:59:24.973256 5855 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:24.962699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45029 (local address 127.0.0.1:50022)
0620 02:59:24.963347 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:24.963362 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:24.963479 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:24.964005 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:24.964009 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:24.964019 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:59:24.964331 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:24.964341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:24.965796 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:24.965800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:24.966857 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:24.966868 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:24.967826 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:24.967888 (+ 62us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:24.968012 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:24.968087 (+ 75us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:59:24.970878 (+ 2791us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37883/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 02:59:24.972832 (+ 1954us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:45029: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37883/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":1651,"thread_start_us":354,"threads_started":1}
I20260620 02:59:24.974557 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 5760
I20260620 02:59:24.975660 5821 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 02:59:24.984021 5856 webserver.cc:533] Webserver: sslize failed:
E20260620 02:59:24.984145 5856 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 02:59:25.067495 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 5760
2026-06-20T02:59:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (754 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T02:59:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:59:25Z Disabled control of system clock
I20260620 02:59:25.148324 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36233
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36233
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924320356866-388-0/testchainca.cert with env {}
W20260620 02:59:25.466635 5861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:25.466914 5861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:25.467000 5861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:25.473837 5861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:59:25.473935 5861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:25.473971 5861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:25.474025 5861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:59:25.474067 5861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:59:25.474119 5861 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:59:25.474159 5861 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 02:59:25.483731 5861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36233
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924320356866-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:25.485028 5861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:25.486330 5861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:25.498042 5866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:25.498082 5867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:25.498258 5869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:25.498965 5861 server_base.cc:1034] running on GCE node
I20260620 02:59:25.499722 5861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:25.501130 5861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:25.502349 5861 hybrid_clock.cc:648] HybridClock initialized: now 1781924365502294 us; error 53 us; skew 500 ppm
I20260620 02:59:25.502693 5861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:59:25.504985 5861 webserver.cc:460] Webserver started at http://127.0.97.62:40561/ using document root <none> and password file <none>
I20260620 02:59:25.505748 5861 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:25.505879 5861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:25.506194 5861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:25.509024 5861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "4466825baace44c586efcb52fc4711fb"
format_stamp: "Formatted at 2026-06-20 02:59:25 on dist-test-slave-f1j0"
server_key: "c2a8e90ba2f2264b46a8d5e635ed9424"
server_key_iv: "a54707a67f9e3a0d5d0a3033561f40fc"
server_key_version: "encryptionkey@0"
I20260620 02:59:25.509876 5861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "4466825baace44c586efcb52fc4711fb"
format_stamp: "Formatted at 2026-06-20 02:59:25 on dist-test-slave-f1j0"
server_key: "c2a8e90ba2f2264b46a8d5e635ed9424"
server_key_iv: "a54707a67f9e3a0d5d0a3033561f40fc"
server_key_version: "encryptionkey@0"
I20260620 02:59:25.514724 5861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:59:25.517992 5875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:25.519275 5861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:59:25.519450 5861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "4466825baace44c586efcb52fc4711fb"
format_stamp: "Formatted at 2026-06-20 02:59:25 on dist-test-slave-f1j0"
server_key: "c2a8e90ba2f2264b46a8d5e635ed9424"
server_key_iv: "a54707a67f9e3a0d5d0a3033561f40fc"
server_key_version: "encryptionkey@0"
I20260620 02:59:25.519675 5861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:25.543009 5861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:25.543883 5861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:25.544173 5861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:25.561808 5861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36233
I20260620 02:59:25.561829 5926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36233 every 8 connection(s)
I20260620 02:59:25.563119 5861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:59:25.566608 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 5861
I20260620 02:59:25.566834 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 02:59:25.566876 5927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:25.567479 388 external_mini_cluster.cc:1428] Setting key e882c32188d80c616c82ffcc1fc7be0e
I20260620 02:59:25.580617 5927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb: Bootstrap starting.
I20260620 02:59:25.580758 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:25.569070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.0.97.62:36233)
0620 02:59:25.570049 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:25.570059 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:25.570106 (+ 47us) server_negotiation.cc:407] Connection header received
0620 02:59:25.571805 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:25.571825 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:25.572334 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:25.572667 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:25.573287 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:25.574786 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:25.576505 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:25.577167 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:25.577311 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:25.578545 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:25.578553 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:25.578567 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:59:25.579115 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:25.579185 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:59:25.579206 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:25.579288 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:25.579509 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 02:59:25.579834 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":356,"threads_started":1}
I20260620 02:59:25.584336 5927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:25.584559 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:59:25.585346 5927 log.cc:824] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:25.588524 5927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb: No bootstrap required, opened a new log
I20260620 02:59:25.595925 5927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4466825baace44c586efcb52fc4711fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36233 } }
I20260620 02:59:25.596316 5927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:25.596426 5927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4466825baace44c586efcb52fc4711fb, State: Initialized, Role: FOLLOWER
I20260620 02:59:25.597014 5927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4466825baace44c586efcb52fc4711fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36233 } }
I20260620 02:59:25.597255 5927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:59:25.597411 5927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:59:25.597621 5927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:25.600013 5927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4466825baace44c586efcb52fc4711fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36233 } }
I20260620 02:59:25.600538 5927 leader_election.cc:302] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4466825baace44c586efcb52fc4711fb; no voters:
I20260620 02:59:25.601110 5927 leader_election.cc:288] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:59:25.601274 5932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:25.601965 5932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb [term 1 LEADER]: Becoming Leader. State: Replica: 4466825baace44c586efcb52fc4711fb, State: Running, Role: LEADER
I20260620 02:59:25.602468 5932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4466825baace44c586efcb52fc4711fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36233 } }
I20260620 02:59:25.603279 5927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:59:25.605536 5934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4466825baace44c586efcb52fc4711fb. Latest consensus state: current_term: 1 leader_uuid: "4466825baace44c586efcb52fc4711fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4466825baace44c586efcb52fc4711fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36233 } } }
I20260620 02:59:25.606029 5934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:25.606180 5933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4466825baace44c586efcb52fc4711fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4466825baace44c586efcb52fc4711fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36233 } } }
I20260620 02:59:25.606611 5933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:25.608105 5938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:59:25.613535 5938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:59:25.623199 5938 catalog_manager.cc:1347] Generated new cluster ID: efbee3f82f854ffd924aa1b50c6c4ba2
I20260620 02:59:25.623411 5938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:59:25.638023 5938 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:59:25.639266 5938 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:59:25.646301 5938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4466825baace44c586efcb52fc4711fb: Generated new TSK 0
I20260620 02:59:25.647087 5938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:59:25.733675 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:25.662676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.0.97.62:36233)
0620 02:59:25.662870 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:25.662877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:25.663367 (+ 490us) server_negotiation.cc:407] Connection header received
0620 02:59:25.663578 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:25.663584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:25.663680 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:25.663871 (+ 191us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:59:25.664299 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:25.665396 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:25.667016 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:25.667711 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:25.667838 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:25.668407 (+ 569us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:59:25.732912 (+ 64505us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 02:59:25.733322 (+ 410us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55340: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 02:59:25.733820 5930 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55340: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 02:59:25.733887 5956 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:25.662501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36233 (local address 127.0.0.1:55340)
0620 02:59:25.663259 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:25.663277 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:25.663403 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:25.663885 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:25.663889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:25.663899 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:59:25.664154 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:25.664163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:25.665644 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:25.665649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:25.666856 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:25.666869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:25.667852 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:25.667900 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:25.668026 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:25.668073 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:59:25.733222 (+ 65149us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 02:59:25.733459 (+ 237us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:36233: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":413,"threads_started":1}
I20260620 02:59:25.735538 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 5861
I20260620 02:59:25.736738 5922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:25.833977 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 5861
2026-06-20T02:59:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (746 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (409 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (401 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:59:26 dist-test-slave-f1j0 krb5kdc[5971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:59:26 dist-test-slave-f1j0 krb5kdc[5971](info): set up 2 sockets
Jun 20 02:59:26 dist-test-slave-f1j0 krb5kdc[5971](info): commencing operation
krb5kdc: starting...
W20260620 02:59:27.714249 388 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 02:59:27 dist-test-slave-f1j0 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:59:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:59:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:59:27.920413 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34987
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:28.242100 5984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:28.242420 5984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:28.242508 5984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:28.249897 5984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:59:28.250015 5984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:28.250061 5984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:28.250109 5984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:59:28.250154 5984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:59:28.260144 5984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:28.261447 5984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:28.262825 5984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:28.274461 5990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:28.274528 5989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:28.274549 5992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:28.275620 5984 server_base.cc:1034] running on GCE node
I20260620 02:59:28.276407 5984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:28.277827 5984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:28.279062 5984 hybrid_clock.cc:648] HybridClock initialized: now 1781924368279024 us; error 32 us; skew 500 ppm
I20260620 02:59:28.279421 5984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:28 dist-test-slave-f1j0 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:28.282500 5984 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:59:28.283967 5984 webserver.cc:460] Webserver started at http://127.0.97.62:40535/ using document root <none> and password file <none>
I20260620 02:59:28.284632 5984 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:28.284760 5984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:28.285080 5984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:28.288547 5984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "c2e79ea3907f4ca485c9d3dd14625c6e"
format_stamp: "Formatted at 2026-06-20 02:59:28 on dist-test-slave-f1j0"
server_key: "ead495d43ef8914e0d699bcacf5b94e4"
server_key_iv: "ede639eaf5f0482de74ec132928e1e24"
server_key_version: "encryptionkey@0"
I20260620 02:59:28.289307 5984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "c2e79ea3907f4ca485c9d3dd14625c6e"
format_stamp: "Formatted at 2026-06-20 02:59:28 on dist-test-slave-f1j0"
server_key: "ead495d43ef8914e0d699bcacf5b94e4"
server_key_iv: "ede639eaf5f0482de74ec132928e1e24"
server_key_version: "encryptionkey@0"
I20260620 02:59:28.294484 5984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:59:28.297953 5999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:28.299310 5984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:59:28.299486 5984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "c2e79ea3907f4ca485c9d3dd14625c6e"
format_stamp: "Formatted at 2026-06-20 02:59:28 on dist-test-slave-f1j0"
server_key: "ead495d43ef8914e0d699bcacf5b94e4"
server_key_iv: "ede639eaf5f0482de74ec132928e1e24"
server_key_version: "encryptionkey@0"
I20260620 02:59:28.299679 5984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:28.312011 5984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:28.316372 5984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:28.316653 5984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:28.334786 5984 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34987
I20260620 02:59:28.334808 6050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34987 every 8 connection(s)
I20260620 02:59:28.336164 5984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:59:28.339515 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 5984
I20260620 02:59:28.339761 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 02:59:28.339931 6051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:28.340412 388 external_mini_cluster.cc:1428] Setting key c0febffe14d2bb642743b1e0e571bece
I20260620 02:59:28.354489 6051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e: Bootstrap starting.
Jun 20 02:59:28 dist-test-slave-f1j0 krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:28.358371 6051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:28.359761 6051 log.cc:824] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:28.362776 6051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e: No bootstrap required, opened a new log
I20260620 02:59:28.364113 6054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:28.342337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38830 (local address 127.0.97.62:34987)
0620 02:59:28.343182 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:28.343196 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:28.343245 (+ 49us) server_negotiation.cc:407] Connection header received
0620 02:59:28.344703 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:28.344726 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:28.345142 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:28.345483 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:28.347275 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:28.348566 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:28.350329 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:28.351037 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:28.351172 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:28.356069 (+ 4897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:28.356082 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:28.356103 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:28.356155 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:28.360076 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:28.360787 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:28.360794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:28.360800 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:28.360979 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:28.361364 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:28.361384 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:28.361386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:28.361970 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:28.362284 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:28.362708 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 02:59:28.363134 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":446,"threads_started":1}
I20260620 02:59:28.370991 6051 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e79ea3907f4ca485c9d3dd14625c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34987 } }
I20260620 02:59:28.371443 6051 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:28.371547 6051 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e79ea3907f4ca485c9d3dd14625c6e, State: Initialized, Role: FOLLOWER
I20260620 02:59:28.372161 6051 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e79ea3907f4ca485c9d3dd14625c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34987 } }
I20260620 02:59:28.372391 6051 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:59:28.372530 6051 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:59:28.372705 6051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:28.374833 6051 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e79ea3907f4ca485c9d3dd14625c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34987 } }
I20260620 02:59:28.375337 6051 leader_election.cc:302] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2e79ea3907f4ca485c9d3dd14625c6e; no voters:
I20260620 02:59:28.375960 6051 leader_election.cc:288] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:59:28.376081 6056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:28.376968 6056 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e [term 1 LEADER]: Becoming Leader. State: Replica: c2e79ea3907f4ca485c9d3dd14625c6e, State: Running, Role: LEADER
I20260620 02:59:28.377487 6056 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e79ea3907f4ca485c9d3dd14625c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34987 } }
I20260620 02:59:28.378247 6051 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:59:28.380338 6058 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2e79ea3907f4ca485c9d3dd14625c6e. Latest consensus state: current_term: 1 leader_uuid: "c2e79ea3907f4ca485c9d3dd14625c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e79ea3907f4ca485c9d3dd14625c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34987 } } }
I20260620 02:59:28.380257 6057 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2e79ea3907f4ca485c9d3dd14625c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e79ea3907f4ca485c9d3dd14625c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34987 } } }
I20260620 02:59:28.380708 6057 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:28.380715 6058 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:28.381932 6063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:59:28.389266 6063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:59:28.398136 6063 catalog_manager.cc:1347] Generated new cluster ID: c6e4d794ff7c4dfba8c1de4cfed85c55
I20260620 02:59:28.398380 6063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:59:28.416682 6063 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:59:28.418095 6063 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:59:28.427632 6063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2e79ea3907f4ca485c9d3dd14625c6e: Generated new TSK 0
I20260620 02:59:28.428509 6063 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:59:28.489555 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34987
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:28.816556 6077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:28.816833 6077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:28.816985 6077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:28.824419 6077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:28.824560 6077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:28.824750 6077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:59:28.834518 6077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:28.835850 6077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:28.837137 6077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:28.849828 6082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:28.850087 6083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:28.850518 6085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:28.850914 6077 server_base.cc:1034] running on GCE node
I20260620 02:59:28.851511 6077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:28.852324 6077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:28.853545 6077 hybrid_clock.cc:648] HybridClock initialized: now 1781924368853510 us; error 49 us; skew 500 ppm
I20260620 02:59:28.853895 6077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:28 dist-test-slave-f1j0 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:28.857707 6077 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:59:28.859616 6077 webserver.cc:460] Webserver started at http://127.0.97.1:33045/ using document root <none> and password file <none>
I20260620 02:59:28.860661 6077 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:28.860800 6077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:28.861194 6077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:28.865240 6077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "da74e769af09480495f491e504062caa"
format_stamp: "Formatted at 2026-06-20 02:59:28 on dist-test-slave-f1j0"
server_key: "bbbbfbabb3fb3d4141766a9503859ed5"
server_key_iv: "b799c3aef63408cfdebbc9d45fe3c395"
server_key_version: "encryptionkey@0"
I20260620 02:59:28.866035 6077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "da74e769af09480495f491e504062caa"
format_stamp: "Formatted at 2026-06-20 02:59:28 on dist-test-slave-f1j0"
server_key: "bbbbfbabb3fb3d4141766a9503859ed5"
server_key_iv: "b799c3aef63408cfdebbc9d45fe3c395"
server_key_version: "encryptionkey@0"
I20260620 02:59:28.871068 6077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:59:28.874348 6092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:28.875638 6077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:59:28.875814 6077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "da74e769af09480495f491e504062caa"
format_stamp: "Formatted at 2026-06-20 02:59:28 on dist-test-slave-f1j0"
server_key: "bbbbfbabb3fb3d4141766a9503859ed5"
server_key_iv: "b799c3aef63408cfdebbc9d45fe3c395"
server_key_version: "encryptionkey@0"
I20260620 02:59:28.876016 6077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:28.908565 6077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:28.912456 6077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:28.912748 6077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:28.913815 6077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:28.915117 6077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:28.915230 6077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:28.915390 6077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:28.915462 6077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:28.947599 6077 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40811
I20260620 02:59:28.947641 6204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40811 every 8 connection(s)
I20260620 02:59:28.948902 6077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:59:28.951821 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 6077
I20260620 02:59:28.952037 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:59:28.952883 388 external_mini_cluster.cc:1428] Setting key 9191d18199d1176b6b5c40bf29afb4ff
Jun 20 02:59:28 dist-test-slave-f1j0 krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:28.968890 6207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:28.951384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55483 (local address 127.0.97.62:34987)
0620 02:59:28.951792 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:28.951796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:28.953002 (+ 1206us) server_negotiation.cc:407] Connection header received
0620 02:59:28.955207 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:28.955212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:28.955298 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:28.955408 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:28.956837 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:28.957522 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:28.959275 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:28.959985 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:28.960120 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:28.963538 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:28.963543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:28.963547 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:28.963578 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:28.965996 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:28.966684 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:28.966692 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:28.966697 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:28.966812 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:28.967233 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:28.967237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:28.967241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:28.967484 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:28.967627 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:28.968328 (+ 701us) server_negotiation.cc:299] Negotiation successful
0620 02:59:28.968536 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":225,"threads_started":1}
I20260620 02:59:28.969966 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:28.952022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34987 (local address 127.0.97.1:55483)
0620 02:59:28.952788 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:28.952823 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:28.954949 (+ 2126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:28.955642 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:28.955650 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:28.956130 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:28.956677 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:28.956693 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:28.957703 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:28.957710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:28.959080 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:28.959097 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:28.960224 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:28.960234 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:28.960490 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:28.961179 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:28.961205 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:28.963321 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:28.966133 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:28.966143 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:28.966160 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:28.966566 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:28.966918 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:28.966924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:28.966928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:28.967118 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:28.967618 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:28.967626 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:28.967960 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 02:59:28.968240 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 02:59:28.968767 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":347,"threads_started":1}
I20260620 02:59:28.971653 6205 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34987
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:59:28.971968 6205 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:28.972689 6205 heartbeater.cc:511] Master 127.0.97.62:34987 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:59:28.975013 6016 ts_manager.cc:194] Registered new tserver with Master: da74e769af09480495f491e504062caa (127.0.97.1:40811)
I20260620 02:59:28.976471 6016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:59:29.001083 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34987
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:29.320987 6211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:29.321278 6211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:29.321446 6211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:29.328948 6211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:29.329100 6211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:29.329311 6211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:59:29.339092 6211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:29.340322 6211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:29.341675 6211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:29.354905 6216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:29.354879 6217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:29.354832 6219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:29.355800 6211 server_base.cc:1034] running on GCE node
I20260620 02:59:29.356383 6211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:29.357426 6211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:29.358690 6211 hybrid_clock.cc:648] HybridClock initialized: now 1781924369358615 us; error 60 us; skew 500 ppm
I20260620 02:59:29.359011 6211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:29 dist-test-slave-f1j0 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:29.362470 6211 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:59:29.364001 6211 webserver.cc:460] Webserver started at http://127.0.97.2:33809/ using document root <none> and password file <none>
I20260620 02:59:29.364820 6211 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:29.364984 6211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:29.365304 6211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:29.368283 6211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "ce9892e195c14582877e44c50093581b"
format_stamp: "Formatted at 2026-06-20 02:59:29 on dist-test-slave-f1j0"
server_key: "208f21dceea82f4e7841a0b2cf8c4a42"
server_key_iv: "624319ba23cc1136678c26ca1bb28c3d"
server_key_version: "encryptionkey@0"
I20260620 02:59:29.369040 6211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ce9892e195c14582877e44c50093581b"
format_stamp: "Formatted at 2026-06-20 02:59:29 on dist-test-slave-f1j0"
server_key: "208f21dceea82f4e7841a0b2cf8c4a42"
server_key_iv: "624319ba23cc1136678c26ca1bb28c3d"
server_key_version: "encryptionkey@0"
I20260620 02:59:29.374212 6211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:59:29.377519 6226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:29.378913 6211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 02:59:29.379067 6211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "ce9892e195c14582877e44c50093581b"
format_stamp: "Formatted at 2026-06-20 02:59:29 on dist-test-slave-f1j0"
server_key: "208f21dceea82f4e7841a0b2cf8c4a42"
server_key_iv: "624319ba23cc1136678c26ca1bb28c3d"
server_key_version: "encryptionkey@0"
I20260620 02:59:29.379242 6211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:29.398967 6211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:29.403340 6211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:29.403656 6211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:29.404769 6211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:29.406126 6211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:29.406196 6211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:29.406332 6211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:29.406383 6211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:29.439349 6211 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39455
I20260620 02:59:29.439386 6338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39455 every 8 connection(s)
I20260620 02:59:29.440588 6211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:59:29.450398 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 6211
I20260620 02:59:29.450605 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:59:29.451116 388 external_mini_cluster.cc:1428] Setting key 0aa50bf6c4820564526b8a98e5a66068
Jun 20 02:59:29 dist-test-slave-f1j0 krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:29.460949 6207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:29.443070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54209 (local address 127.0.97.62:34987)
0620 02:59:29.443282 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:29.443289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:29.444410 (+ 1121us) server_negotiation.cc:407] Connection header received
0620 02:59:29.446135 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:29.446139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:29.446205 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:29.446301 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:29.447697 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:29.448405 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:29.449943 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:29.450594 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:29.450738 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:29.454308 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:29.454318 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:29.454326 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:29.454371 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:29.457919 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:29.458629 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:29.458637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:29.458643 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:29.458790 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:29.459203 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:29.459218 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:29.459225 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:29.459530 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:29.459633 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:29.460547 (+ 914us) server_negotiation.cc:299] Negotiation successful
0620 02:59:29.460708 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:59:29.462541 6341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:29.443550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34987 (local address 127.0.97.2:54209)
0620 02:59:29.444255 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:29.444294 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:29.445890 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:29.446500 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:29.446508 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:29.447007 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:29.447558 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:29.447574 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:29.448523 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:29.448527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:29.449825 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:29.449836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:29.450831 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:29.450837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:29.451029 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:29.451744 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:29.451796 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:29.454075 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:29.458050 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:29.458057 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:29.458071 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:29.458499 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:29.458901 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:29.458905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:29.458907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:29.459083 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:29.459653 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:29.459673 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:29.460146 (+ 473us) client_negotiation.cc:769] Sending connection context
0620 02:59:29.460485 (+ 339us) client_negotiation.cc:240] Negotiation successful
0620 02:59:29.461060 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":286,"threads_started":1}
I20260620 02:59:29.463934 6339 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34987
I20260620 02:59:29.464236 6339 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:29.464941 6339 heartbeater.cc:511] Master 127.0.97.62:34987 requested a full tablet report, sending...
I20260620 02:59:29.466325 6016 ts_manager.cc:194] Registered new tserver with Master: ce9892e195c14582877e44c50093581b (127.0.97.2:39455)
I20260620 02:59:29.467104 6016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54209
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:59:29.497008 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34987
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:29.829741 6344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:29.830029 6344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:29.830195 6344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:29.837230 6344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:29.837332 6344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:29.837536 6344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:59:29.846925 6344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:29.848225 6344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:29.849519 6344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:29.862412 6350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:29.862358 6352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:29.862634 6349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:29.863466 6344 server_base.cc:1034] running on GCE node
I20260620 02:59:29.864006 6344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:29.864948 6344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:29.866222 6344 hybrid_clock.cc:648] HybridClock initialized: now 1781924369866163 us; error 68 us; skew 500 ppm
I20260620 02:59:29.866565 6344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:29 dist-test-slave-f1j0 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:29.872136 6344 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:59:29.873494 6344 webserver.cc:460] Webserver started at http://127.0.97.3:34607/ using document root <none> and password file <none>
I20260620 02:59:29.874233 6344 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:29.874331 6344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:29.874583 6344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:29.877545 6344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "128b28f4496344cc84534e86c473b304"
format_stamp: "Formatted at 2026-06-20 02:59:29 on dist-test-slave-f1j0"
server_key: "67f52cbc67451b76e87dfd57fc5095b5"
server_key_iv: "685cee1f5b19d3fffb24b87bd1a394fc"
server_key_version: "encryptionkey@0"
I20260620 02:59:29.878341 6344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "128b28f4496344cc84534e86c473b304"
format_stamp: "Formatted at 2026-06-20 02:59:29 on dist-test-slave-f1j0"
server_key: "67f52cbc67451b76e87dfd57fc5095b5"
server_key_iv: "685cee1f5b19d3fffb24b87bd1a394fc"
server_key_version: "encryptionkey@0"
I20260620 02:59:29.884193 6344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:59:29.887646 6359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:29.888957 6344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:59:29.889083 6344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "128b28f4496344cc84534e86c473b304"
format_stamp: "Formatted at 2026-06-20 02:59:29 on dist-test-slave-f1j0"
server_key: "67f52cbc67451b76e87dfd57fc5095b5"
server_key_iv: "685cee1f5b19d3fffb24b87bd1a394fc"
server_key_version: "encryptionkey@0"
I20260620 02:59:29.889251 6344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:29.938325 6344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:29.942746 6344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:29.943043 6344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:29.943982 6344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:29.945283 6344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:29.945386 6344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:29.945531 6344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:29.945631 6344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 02:59:29.977991 6344 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33103
I20260620 02:59:29.978072 6471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33103 every 8 connection(s)
I20260620 02:59:29.979434 6344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:59:29.979665 6205 heartbeater.cc:503] Master 127.0.97.62:34987 was elected leader, sending a full tablet report...
I20260620 02:59:29.986373 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 6344
I20260620 02:59:29.986570 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:59:29.986971 388 external_mini_cluster.cc:1428] Setting key 4ddf06964d6f315cc257d77dd67abf9f
Jun 20 02:59:29 dist-test-slave-f1j0 krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:29.999094 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:29.982304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51827 (local address 127.0.97.62:34987)
0620 02:59:29.982760 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:29.982767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:29.983995 (+ 1228us) server_negotiation.cc:407] Connection header received
0620 02:59:29.985836 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:29.985840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:29.985927 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:29.986038 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:29.987719 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:29.988530 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:29.990133 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:29.990514 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:29.990653 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:29.994173 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:29.994178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:29.994181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:29.994214 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:29.996535 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:29.997155 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:29.997158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:29.997161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:29.997286 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:29.997692 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:29.997695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:29.997697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:29.997962 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:29.998070 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:29.998673 (+ 603us) server_negotiation.cc:299] Negotiation successful
0620 02:59:29.998837 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":239,"threads_started":1}
I20260620 02:59:30.000058 6475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:29.983001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34987 (local address 127.0.97.3:51827)
0620 02:59:29.983838 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:29.983889 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:29.985506 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:29.986291 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:29.986299 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:29.986919 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:29.987550 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:29.987568 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:29.988675 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:29.988680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:29.989978 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:29.989989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:29.990734 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:29.990742 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:29.990963 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:29.991580 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:29.991603 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:29.993980 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:29.996648 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:29.996654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:29.996667 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:29.997044 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:29.997374 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:29.997377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:29.997379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:29.997537 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:29.998065 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:29.998071 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:29.998369 (+ 298us) client_negotiation.cc:769] Sending connection context
0620 02:59:29.998586 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 02:59:29.998943 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":360,"threads_started":1}
I20260620 02:59:30.001782 6472 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34987
I20260620 02:59:30.002140 6472 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:30.002935 6472 heartbeater.cc:511] Master 127.0.97.62:34987 requested a full tablet report, sending...
I20260620 02:59:30.004767 6015 ts_manager.cc:194] Registered new tserver with Master: 128b28f4496344cc84534e86c473b304 (127.0.97.3:33103)
I20260620 02:59:30.005795 6015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51827
I20260620 02:59:30.015414 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/test-admin.keytab.
Jun 20 02:59:30 dist-test-slave-f1j0 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:30.043769 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 02:59:30.053467 6480 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:30.050945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54636 (local address 127.0.97.1:40811)
0620 02:59:30.051562 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:30.051570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:30.051599 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:59:30.051759 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:30.051767 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:30.051989 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:30.052125 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:30.053044 (+ 919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54636: BlockingRecv error: recv got EOF from 127.0.0.1:54636 (error 108)
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":295,"threads_started":1}
W20260620 02:59:30.053895 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 02:59:30 dist-test-slave-f1j0 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:30.055414 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 02:59:30 dist-test-slave-f1j0 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:30.056908 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 02:59:30 dist-test-slave-f1j0 krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:59:30.072221 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:30.058700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54652 (local address 127.0.97.1:40811)
0620 02:59:30.058894 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:30.058900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:30.059101 (+ 201us) server_negotiation.cc:407] Connection header received
0620 02:59:30.059279 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:30.059284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:30.059341 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:30.059422 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:30.060378 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:30.061328 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:30.062506 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:30.063041 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:30.063141 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:30.066868 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:30.066879 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:30.066893 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:30.066925 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:30.069733 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:30.070319 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:30.070327 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:30.070329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:30.070440 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:30.070756 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:30.070759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:30.070761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:30.071093 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:30.071308 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:30.071610 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 02:59:30.071801 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 02:59:30.077844 6480 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:30.076080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54656 (local address 127.0.97.1:40811)
0620 02:59:30.076248 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:30.076252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:30.076664 (+ 412us) server_negotiation.cc:407] Connection header received
0620 02:59:30.076782 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:30.076786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:30.076837 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:30.076911 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:30.077529 (+ 618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54656: BlockingRecv error: recv got EOF from 127.0.0.1:54656 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 02:59:30.078109 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 02:59:30 dist-test-slave-f1j0 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:30.079698 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 02:59:30 dist-test-slave-f1j0 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:30.081037 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 02:59:30 dist-test-slave-f1j0 krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:59:30.094802 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:30.082464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54664 (local address 127.0.97.1:40811)
0620 02:59:30.082642 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:30.082647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:30.082900 (+ 253us) server_negotiation.cc:407] Connection header received
0620 02:59:30.083062 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:30.083067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:30.083132 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:30.083251 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:30.084121 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:30.084811 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:30.086105 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:30.086528 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:30.086667 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:30.090063 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:30.090068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:30.090072 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:30.090118 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:30.092532 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:30.093204 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:30.093207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:30.093210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:30.093322 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:30.093738 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:30.093742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:30.093744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:30.093996 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:30.094195 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:30.094421 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 02:59:30.094571 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 02:59:30.099145 6480 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:30.097463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54674 (local address 127.0.97.1:40811)
0620 02:59:30.097670 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:30.097677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:30.097865 (+ 188us) server_negotiation.cc:407] Connection header received
0620 02:59:30.098042 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:30.098047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:30.098155 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:30.098256 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:30.098900 (+ 644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54674: BlockingRecv error: recv got EOF from 127.0.0.1:54674 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 02:59:30.099325 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781924320356866-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 02:59:30 dist-test-slave-f1j0 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:30.100776 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 02:59:30 dist-test-slave-f1j0 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:30.102030 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 02:59:30 dist-test-slave-f1j0 krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:59:30.117069 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:30.103150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54682 (local address 127.0.97.1:40811)
0620 02:59:30.103339 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:30.103346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:30.103629 (+ 283us) server_negotiation.cc:407] Connection header received
0620 02:59:30.103805 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:30.103810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:30.103871 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:30.103965 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:30.104873 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:30.105552 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:30.106772 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:30.107174 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:30.107294 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:30.110520 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:30.110525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:30.110528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:30.110555 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:30.113804 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:30.114637 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:30.114646 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:30.114652 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:30.114842 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:30.115330 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:30.115338 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:30.115342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:30.115721 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:30.116072 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:30.116378 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 02:59:30.116636 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:59:30.118881 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 6077
I20260620 02:59:30.130662 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:30.119591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54696 (local address 127.0.97.1:40811)
0620 02:59:30.120084 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:30.120089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:30.120113 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:59:30.120261 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:30.120265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:30.120316 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:30.120387 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:30.121594 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:30.122528 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:30.123679 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:30.124077 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:30.124179 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:30.126114 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:30.126119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:30.126124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:30.126150 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:30.128326 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:30.128878 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:30.128881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:30.128884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:30.128996 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:30.129322 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:30.129326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:30.129328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:30.129690 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:30.129896 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:30.130127 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 02:59:30.130327 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362}
I20260620 02:59:30.131248 6200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:30.213752 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 6077
I20260620 02:59:30.229712 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 6211
Jun 20 02:59:30 dist-test-slave-f1j0 krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:59:30.244534 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:30.230674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38670 (local address 127.0.97.2:39455)
0620 02:59:30.231273 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:30.231279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:30.231305 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:59:30.231371 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:30.231377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:30.231584 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:30.231717 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:30.232608 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:30.233745 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:30.234896 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:30.235432 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:30.235526 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:30.239106 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:30.239117 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:30.239157 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:30.239193 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:30.241877 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:30.242575 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:30.242583 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:30.242585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:30.242723 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:30.243175 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:30.243179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:30.243181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:30.243511 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:30.243734 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:30.244059 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 02:59:30.244256 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":319,"threads_started":1}
I20260620 02:59:30.245958 6334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:30.327239 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 6211
I20260620 02:59:30.342658 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 6344
Jun 20 02:59:30 dist-test-slave-f1j0 krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:59:30.357949 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:30.343722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47570 (local address 127.0.97.3:33103)
0620 02:59:30.344345 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:30.344352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:30.344379 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:59:30.344438 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:30.344444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:30.344648 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:30.344778 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:30.345820 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:30.346805 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:30.348139 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:30.348696 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:30.348829 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:30.352732 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:30.352743 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:30.352760 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:30.352792 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:30.355335 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:30.355971 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:30.355978 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:30.355981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:30.356096 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:30.356454 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:30.356461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:30.356464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:30.356801 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:30.357106 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:30.357395 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 02:59:30.357660 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":324,"threads_started":1}
I20260620 02:59:30.359184 6467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:30.441464 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 6344
I20260620 02:59:30.457171 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 5984
I20260620 02:59:30.458431 6046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:30.559079 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 5984
2026-06-20T02:59:30Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3919 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:59:30 dist-test-slave-f1j0 krb5kdc[6500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:59:30 dist-test-slave-f1j0 krb5kdc[6500](info): set up 2 sockets
Jun 20 02:59:30 dist-test-slave-f1j0 krb5kdc[6500](info): commencing operation
krb5kdc: starting...
W20260620 02:59:31.638785 388 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 02:59:31 dist-test-slave-f1j0 krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:59:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:59:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 02:59:31.841451 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35199
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:32.158144 6513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:32.158421 6513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:32.158596 6513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:32.165591 6513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:59:32.165693 6513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:32.165740 6513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:32.165849 6513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:59:32.165925 6513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:59:32.175407 6513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:32.176748 6513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:32.178078 6513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:32.189488 6519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:32.189458 6521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:32.189534 6518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:32.190303 6513 server_base.cc:1034] running on GCE node
I20260620 02:59:32.191010 6513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:32.192497 6513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:32.193666 6513 hybrid_clock.cc:648] HybridClock initialized: now 1781924372193598 us; error 49 us; skew 500 ppm
I20260620 02:59:32.194010 6513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:32 dist-test-slave-f1j0 krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:32.197089 6513 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 02:59:32.198463 6513 webserver.cc:460] Webserver started at http://127.0.97.62:36705/ using document root <none> and password file <none>
I20260620 02:59:32.199069 6513 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:32.199162 6513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:32.199429 6513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:32.202406 6513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4"
format_stamp: "Formatted at 2026-06-20 02:59:32 on dist-test-slave-f1j0"
server_key: "3559a5f676da4fbb194007045f59b01d"
server_key_iv: "5f86850f9e02f3d75f2735ecf633e729"
server_key_version: "encryptionkey@0"
I20260620 02:59:32.203076 6513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4"
format_stamp: "Formatted at 2026-06-20 02:59:32 on dist-test-slave-f1j0"
server_key: "3559a5f676da4fbb194007045f59b01d"
server_key_iv: "5f86850f9e02f3d75f2735ecf633e729"
server_key_version: "encryptionkey@0"
I20260620 02:59:32.207901 6513 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:59:32.211256 6528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:32.212508 6513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 02:59:32.212688 6513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4"
format_stamp: "Formatted at 2026-06-20 02:59:32 on dist-test-slave-f1j0"
server_key: "3559a5f676da4fbb194007045f59b01d"
server_key_iv: "5f86850f9e02f3d75f2735ecf633e729"
server_key_version: "encryptionkey@0"
I20260620 02:59:32.212877 6513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:32.229151 6513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:32.233527 6513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:32.233906 6513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:32.252205 6513 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35199
I20260620 02:59:32.252235 6579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35199 every 8 connection(s)
I20260620 02:59:32.253664 6513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:59:32.257395 6580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:32.260720 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 6513
I20260620 02:59:32.260957 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 02:59:32.261610 388 external_mini_cluster.cc:1428] Setting key 1f738fdc5cf06591336a2d2e75739a37
I20260620 02:59:32.269997 6580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4: Bootstrap starting.
I20260620 02:59:32.273666 6580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:32.274979 6580 log.cc:824] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:59:32 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:59:32.278261 6580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4: No bootstrap required, opened a new log
I20260620 02:59:32.285017 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:32.263552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.0.97.62:35199)
0620 02:59:32.264314 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:32.264326 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:32.264367 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:59:32.265950 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:32.265972 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:32.266372 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:32.266688 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:32.268473 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:32.269795 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:32.271417 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:32.272077 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:32.272214 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:32.276624 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:32.276634 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:32.276652 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:32.276691 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:32.280588 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:32.281264 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:32.281276 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:32.281286 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:32.281487 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:32.281886 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:32.281891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:32.281895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:32.282574 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:32.282862 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:32.283204 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 02:59:32.283635 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":362,"threads_started":1}
I20260620 02:59:32.287963 6580 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } }
I20260620 02:59:32.288311 6580 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:32.288380 6580 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e5f36f1afb417f86b54cb6c0ee43a4, State: Initialized, Role: FOLLOWER
I20260620 02:59:32.288920 6580 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } }
I20260620 02:59:32.289105 6580 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:59:32.289191 6580 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:59:32.289302 6580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:32.291376 6580 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } }
I20260620 02:59:32.291806 6580 leader_election.cc:302] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2e5f36f1afb417f86b54cb6c0ee43a4; no voters:
I20260620 02:59:32.292352 6580 leader_election.cc:288] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:59:32.292513 6585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:32.293282 6585 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [term 1 LEADER]: Becoming Leader. State: Replica: c2e5f36f1afb417f86b54cb6c0ee43a4, State: Running, Role: LEADER
I20260620 02:59:32.293872 6585 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } }
I20260620 02:59:32.294646 6580 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:59:32.296859 6586 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } }
I20260620 02:59:32.297217 6586 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:32.296939 6587 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2e5f36f1afb417f86b54cb6c0ee43a4. Latest consensus state: current_term: 1 leader_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } }
I20260620 02:59:32.297339 6587 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:32.300921 6591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:59:32.306473 6591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:59:32.315415 6591 catalog_manager.cc:1347] Generated new cluster ID: 437db21f6b864dab8a24edfc3a3cdbeb
I20260620 02:59:32.315521 6591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:59:32.337994 6591 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:59:32.339358 6591 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:59:32.347219 6591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4: Generated new TSK 0
I20260620 02:59:32.348063 6591 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 02:59:32.410769 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35199
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:32.732057 6606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:32.732364 6606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:32.732537 6606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:32.739740 6606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:32.739904 6606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:32.740080 6606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:59:32.750244 6606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:32.751538 6606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:32.752810 6606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:32.766201 6612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:32.766106 6611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:32.766240 6614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:32.766981 6606 server_base.cc:1034] running on GCE node
I20260620 02:59:32.767666 6606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:32.768668 6606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:32.769913 6606 hybrid_clock.cc:648] HybridClock initialized: now 1781924372769857 us; error 45 us; skew 500 ppm
I20260620 02:59:32.770280 6606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:32 dist-test-slave-f1j0 krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:32.773785 6606 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 02:59:32.775269 6606 webserver.cc:460] Webserver started at http://127.0.97.1:34107/ using document root <none> and password file <none>
I20260620 02:59:32.775998 6606 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:32.776101 6606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:32.776353 6606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:32.779311 6606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "3dc18659e29e43eea11d6a12dd573c8d"
format_stamp: "Formatted at 2026-06-20 02:59:32 on dist-test-slave-f1j0"
server_key: "416e0fcfd0cbc0036a22edda7e5bd6b1"
server_key_iv: "7e290780d741c2649f7bca95c59cf0eb"
server_key_version: "encryptionkey@0"
I20260620 02:59:32.780124 6606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3dc18659e29e43eea11d6a12dd573c8d"
format_stamp: "Formatted at 2026-06-20 02:59:32 on dist-test-slave-f1j0"
server_key: "416e0fcfd0cbc0036a22edda7e5bd6b1"
server_key_iv: "7e290780d741c2649f7bca95c59cf0eb"
server_key_version: "encryptionkey@0"
I20260620 02:59:32.785503 6606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:59:32.788919 6621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:32.790310 6606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:59:32.790488 6606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "3dc18659e29e43eea11d6a12dd573c8d"
format_stamp: "Formatted at 2026-06-20 02:59:32 on dist-test-slave-f1j0"
server_key: "416e0fcfd0cbc0036a22edda7e5bd6b1"
server_key_iv: "7e290780d741c2649f7bca95c59cf0eb"
server_key_version: "encryptionkey@0"
I20260620 02:59:32.790680 6606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:32.803496 6606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:32.807814 6606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:32.808120 6606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:32.809047 6606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:32.810416 6606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:32.810508 6606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:32.810653 6606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:32.810717 6606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:32.843417 6606 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43727
I20260620 02:59:32.843452 6733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43727 every 8 connection(s)
I20260620 02:59:32.844664 6606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:59:32.852512 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 6606
I20260620 02:59:32.852727 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:59:32.853442 388 external_mini_cluster.cc:1428] Setting key 6b4425e5fae1ea294008c7f05471fc9b
Jun 20 02:59:32 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:59:32.865285 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:32.847151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60821 (local address 127.0.97.62:35199)
0620 02:59:32.847689 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:32.847695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:32.848615 (+ 920us) server_negotiation.cc:407] Connection header received
0620 02:59:32.850387 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:32.850391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:32.850461 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:32.850558 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:32.851932 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:32.853021 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:32.854775 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:32.855269 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:32.855393 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:32.859207 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:32.859220 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:32.859223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:32.859255 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:32.861805 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:32.862725 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:32.862733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:32.862738 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:32.862877 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:32.863268 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:32.863276 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:32.863282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:32.863663 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:32.863819 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:32.864498 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 02:59:32.864754 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":419,"thread_start_us":236,"threads_started":1}
I20260620 02:59:32.865746 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:32.847782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.1:60821)
0620 02:59:32.848449 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:32.848491 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:32.850146 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:32.850758 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:32.850766 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:32.851235 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:32.851788 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:32.851805 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:32.853175 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:32.853178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:32.854633 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:32.854644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:32.855463 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:32.855467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:32.855662 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:32.856315 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:32.856336 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:32.858953 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:32.861942 (+ 2989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:32.861953 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:32.861971 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:32.862595 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:32.862981 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:32.862984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:32.862986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:32.863152 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:32.863781 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:32.863786 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:32.864072 (+ 286us) client_negotiation.cc:769] Sending connection context
0620 02:59:32.864300 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 02:59:32.864655 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":261,"threads_started":1}
I20260620 02:59:32.867202 6734 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35199
I20260620 02:59:32.867518 6734 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:32.868379 6734 heartbeater.cc:511] Master 127.0.97.62:35199 requested a full tablet report, sending...
I20260620 02:59:32.870522 6545 ts_manager.cc:194] Registered new tserver with Master: 3dc18659e29e43eea11d6a12dd573c8d (127.0.97.1:43727)
I20260620 02:59:32.872044 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60821
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 02:59:32.900977 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35199
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:33.222838 6740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:33.223183 6740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:33.223356 6740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:33.230648 6740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:33.230801 6740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:33.230983 6740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:59:33.240731 6740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:33.242086 6740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:33.243419 6740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:33.256186 6746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:33.256294 6748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:33.256920 6745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:33.257637 6740 server_base.cc:1034] running on GCE node
I20260620 02:59:33.258247 6740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:33.259222 6740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:33.260433 6740 hybrid_clock.cc:648] HybridClock initialized: now 1781924373260382 us; error 45 us; skew 500 ppm
I20260620 02:59:33.260743 6740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:33 dist-test-slave-f1j0 krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924373, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:33.263950 6740 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 02:59:33.265539 6740 webserver.cc:460] Webserver started at http://127.0.97.2:45041/ using document root <none> and password file <none>
I20260620 02:59:33.266258 6740 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:33.266402 6740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:33.266709 6740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:33.269529 6740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "0fe8419e2d0a4217880d7b0ca459c073"
format_stamp: "Formatted at 2026-06-20 02:59:33 on dist-test-slave-f1j0"
server_key: "8f5310ac57179c6fc395f6218cf443a1"
server_key_iv: "246844326046e5e598b68d36801d0f2e"
server_key_version: "encryptionkey@0"
I20260620 02:59:33.270368 6740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0fe8419e2d0a4217880d7b0ca459c073"
format_stamp: "Formatted at 2026-06-20 02:59:33 on dist-test-slave-f1j0"
server_key: "8f5310ac57179c6fc395f6218cf443a1"
server_key_iv: "246844326046e5e598b68d36801d0f2e"
server_key_version: "encryptionkey@0"
I20260620 02:59:33.275319 6740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:59:33.278661 6755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:33.280030 6740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:59:33.280232 6740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "0fe8419e2d0a4217880d7b0ca459c073"
format_stamp: "Formatted at 2026-06-20 02:59:33 on dist-test-slave-f1j0"
server_key: "8f5310ac57179c6fc395f6218cf443a1"
server_key_iv: "246844326046e5e598b68d36801d0f2e"
server_key_version: "encryptionkey@0"
I20260620 02:59:33.280424 6740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:33.309914 6740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:33.314177 6740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:33.314472 6740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:33.315430 6740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:33.316929 6740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:33.317029 6740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:33.317203 6740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:33.317284 6740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:33.350226 6740 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44039
I20260620 02:59:33.350318 6867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44039 every 8 connection(s)
I20260620 02:59:33.351531 6740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:59:33.360666 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 6740
I20260620 02:59:33.360865 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:59:33.361428 388 external_mini_cluster.cc:1428] Setting key a5793a867d3db645e9bfdc0ba6de698b
Jun 20 02:59:33 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924373, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:59:33.374425 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:33.354142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33345 (local address 127.0.97.62:35199)
0620 02:59:33.354320 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:33.354325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:33.355489 (+ 1164us) server_negotiation.cc:407] Connection header received
0620 02:59:33.357285 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:33.357290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:33.357379 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:33.357492 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:33.359006 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:33.359732 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:33.362056 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:33.362639 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:33.362740 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:33.366863 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:33.366874 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:33.366882 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:33.366932 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:33.370833 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:33.371908 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:33.371913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:33.371915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:33.372051 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:33.372655 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:33.372659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:33.372661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:33.372927 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:33.373044 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:33.373992 (+ 948us) server_negotiation.cc:299] Negotiation successful
0620 02:59:33.374173 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:59:33.376036 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:33.354665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.2:33345)
0620 02:59:33.355322 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:33.355366 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:33.357005 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:33.357764 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:33.357773 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:33.358263 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:33.358825 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:33.358842 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:33.359891 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:33.359901 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:33.361877 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:33.361892 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:33.362883 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:33.362891 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:33.363185 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:33.363921 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:33.363944 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:33.366610 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:33.371012 (+ 4402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:33.371026 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:33.371047 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:33.371748 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:33.372171 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:33.372179 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:33.372185 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:33.372457 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:33.373061 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:33.373072 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:33.373591 (+ 519us) client_negotiation.cc:769] Sending connection context
0620 02:59:33.373904 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 02:59:33.374526 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":269,"threads_started":1}
I20260620 02:59:33.377558 6868 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35199
I20260620 02:59:33.377928 6868 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:33.378631 6868 heartbeater.cc:511] Master 127.0.97.62:35199 requested a full tablet report, sending...
I20260620 02:59:33.380097 6545 ts_manager.cc:194] Registered new tserver with Master: 0fe8419e2d0a4217880d7b0ca459c073 (127.0.97.2:44039)
I20260620 02:59:33.380895 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33345
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 02:59:33.412482 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35199
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:33.736696 6873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:33.737025 6873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:33.737196 6873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:33.744594 6873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:33.744748 6873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:33.744922 6873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:59:33.754631 6873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:33.755990 6873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:33.757329 6873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:33.770277 6878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:33.770318 6879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:33.770349 6881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:33.771189 6873 server_base.cc:1034] running on GCE node
I20260620 02:59:33.771728 6873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:33.772645 6873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:33.773882 6873 hybrid_clock.cc:648] HybridClock initialized: now 1781924373773813 us; error 56 us; skew 500 ppm
I20260620 02:59:33.774199 6873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:33 dist-test-slave-f1j0 krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924373, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:33.777412 6873 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 02:59:33.778975 6873 webserver.cc:460] Webserver started at http://127.0.97.3:38173/ using document root <none> and password file <none>
I20260620 02:59:33.779670 6873 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:33.779807 6873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:33.780118 6873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:33.783062 6873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "f27e01b9b2cc4d48a7caddbdce674145"
format_stamp: "Formatted at 2026-06-20 02:59:33 on dist-test-slave-f1j0"
server_key: "7a56acfdd3dd51f80463cb6b8a7d6783"
server_key_iv: "61523f33cfbd436ae2494f7f9ad2165b"
server_key_version: "encryptionkey@0"
I20260620 02:59:33.783931 6873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f27e01b9b2cc4d48a7caddbdce674145"
format_stamp: "Formatted at 2026-06-20 02:59:33 on dist-test-slave-f1j0"
server_key: "7a56acfdd3dd51f80463cb6b8a7d6783"
server_key_iv: "61523f33cfbd436ae2494f7f9ad2165b"
server_key_version: "encryptionkey@0"
I20260620 02:59:33.789450 6873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:59:33.792886 6888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:33.794348 6873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:59:33.794515 6873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "f27e01b9b2cc4d48a7caddbdce674145"
format_stamp: "Formatted at 2026-06-20 02:59:33 on dist-test-slave-f1j0"
server_key: "7a56acfdd3dd51f80463cb6b8a7d6783"
server_key_iv: "61523f33cfbd436ae2494f7f9ad2165b"
server_key_version: "encryptionkey@0"
I20260620 02:59:33.794725 6873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:33.810483 6873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:33.814800 6873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:33.815112 6873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:33.816042 6873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:33.817394 6873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:33.817503 6873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:33.817660 6873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:33.817734 6873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:33.850970 6873 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44065
I20260620 02:59:33.851016 7000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44065 every 8 connection(s)
I20260620 02:59:33.852187 6873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:59:33.862607 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 6873
I20260620 02:59:33.862825 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:59:33.863376 388 external_mini_cluster.cc:1428] Setting key 507c86d7f9f77bd22e49e141a0574da9
Jun 20 02:59:33 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924373, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:59:33.874514 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:33.854759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52463 (local address 127.0.97.62:35199)
0620 02:59:33.854930 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:33.854934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:33.855965 (+ 1031us) server_negotiation.cc:407] Connection header received
0620 02:59:33.857788 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:33.857793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:33.857881 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:33.857993 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:33.859407 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:33.860169 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:33.861760 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:33.862141 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:33.862285 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:33.867183 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:33.867189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:33.867194 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:33.867236 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:33.870855 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:33.871813 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:33.871819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:33.871822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:33.871981 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:33.872605 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:33.872610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:33.872614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:33.872923 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:33.873084 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:33.873935 (+ 851us) server_negotiation.cc:299] Negotiation successful
0620 02:59:33.874159 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 02:59:33.875603 7003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:33.855136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.3:52463)
0620 02:59:33.855814 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:33.855849 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:33.857469 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:33.858218 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:33.858225 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:33.858705 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:33.859239 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:33.859256 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:33.860329 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:33.860333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:33.861539 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:33.861551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:33.862607 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:33.862614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:33.862920 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:33.863840 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:33.863869 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:33.866988 (+ 3119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:33.871035 (+ 4047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:33.871041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:33.871055 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:33.871661 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:33.872099 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:33.872104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:33.872108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:33.872439 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:33.873058 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:33.873114 (+ 56us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:33.873486 (+ 372us) client_negotiation.cc:769] Sending connection context
0620 02:59:33.873834 (+ 348us) client_negotiation.cc:240] Negotiation successful
0620 02:59:33.874388 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":263,"threads_started":1}
I20260620 02:59:33.875708 6734 heartbeater.cc:503] Master 127.0.97.62:35199 was elected leader, sending a full tablet report...
I20260620 02:59:33.877341 7001 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35199
I20260620 02:59:33.877696 7001 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:33.878556 7001 heartbeater.cc:511] Master 127.0.97.62:35199 requested a full tablet report, sending...
I20260620 02:59:33.879988 6545 ts_manager.cc:194] Registered new tserver with Master: f27e01b9b2cc4d48a7caddbdce674145 (127.0.97.3:44065)
I20260620 02:59:33.880826 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52463
I20260620 02:59:33.891741 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260620 02:59:33.938861 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43161
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35199,127.0.97.61:43161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:34.264084 7006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:34.264364 7006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:34.264518 7006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:34.271718 7006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:59:34.271834 7006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:34.271929 7006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:34.272003 7006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:59:34.272092 7006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:59:34.282019 7006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35199,127.0.97.61:43161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
W20260620 02:59:34.282377 7006 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:35199,127.0.97.61:43161'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 02:59:34.283466 7006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:34.284865 7006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:34.296608 7012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:34.296696 7011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:34.296653 7014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:34.302822 7006 server_base.cc:1034] running on GCE node
I20260620 02:59:34.303622 7006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:34.305038 7006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:34.306286 7006 hybrid_clock.cc:648] HybridClock initialized: now 1781924374306204 us; error 59 us; skew 500 ppm
I20260620 02:59:34.306638 7006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:34 dist-test-slave-f1j0 krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924374, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:34.310284 7006 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 02:59:34.311811 7006 webserver.cc:460] Webserver started at http://127.0.97.61:36701/ using document root <none> and password file <none>
I20260620 02:59:34.312501 7006 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:34.312587 7006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:34.312835 7006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:34.315966 7006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/data/instance:
uuid: "683840b125cc4cf791dc563ae6126dc9"
format_stamp: "Formatted at 2026-06-20 02:59:34 on dist-test-slave-f1j0"
server_key: "45123c0c83027f5756e8846ef116083a"
server_key_iv: "79b2e5e4b2c070eb07efeb363b8b2ab6"
server_key_version: "encryptionkey@0"
I20260620 02:59:34.316830 7006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/wal/instance:
uuid: "683840b125cc4cf791dc563ae6126dc9"
format_stamp: "Formatted at 2026-06-20 02:59:34 on dist-test-slave-f1j0"
server_key: "45123c0c83027f5756e8846ef116083a"
server_key_iv: "79b2e5e4b2c070eb07efeb363b8b2ab6"
server_key_version: "encryptionkey@0"
I20260620 02:59:34.322232 7006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 02:59:34.325891 7021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:34.327280 7006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 02:59:34.327484 7006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/wal
uuid: "683840b125cc4cf791dc563ae6126dc9"
format_stamp: "Formatted at 2026-06-20 02:59:34 on dist-test-slave-f1j0"
server_key: "45123c0c83027f5756e8846ef116083a"
server_key_iv: "79b2e5e4b2c070eb07efeb363b8b2ab6"
server_key_version: "encryptionkey@0"
I20260620 02:59:34.327718 7006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:34.345275 7006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:34.350332 7006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:34.350621 7006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:34.369234 7006 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:43161
I20260620 02:59:34.369270 7072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:43161 every 8 connection(s)
I20260620 02:59:34.370592 7006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/data/info.pb
I20260620 02:59:34.374217 7073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:34.378546 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 7006
I20260620 02:59:34.383489 6868 heartbeater.cc:503] Master 127.0.97.62:35199 was elected leader, sending a full tablet report...
I20260620 02:59:34.381965 7073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:59:34 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924374, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 02:59:34 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 02:59:34.401353 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:34.386094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60140 (local address 127.0.97.62:35199)
0620 02:59:34.386583 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:34.386591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:34.387283 (+ 692us) server_negotiation.cc:407] Connection header received
0620 02:59:34.387453 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:34.387456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:34.387525 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:34.387609 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:34.389028 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.389835 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.391207 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.391582 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.391715 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:34.395230 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:34.395237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:34.395242 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:34.395282 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:34.398296 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:34.399230 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:34.399236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:34.399240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:34.399368 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:34.399783 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:34.399788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:34.399791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:34.400088 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:34.400246 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:34.400794 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 02:59:34.401004 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":234,"threads_started":1}
I20260620 02:59:34.402482 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:34.386264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.0.1:60140)
0620 02:59:34.387131 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:34.387179 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:34.387341 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:34.387704 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:34.387711 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:34.388522 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:34.388871 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:34.388887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.389956 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.389964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:34.391067 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:34.391078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.391794 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.391800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:34.391946 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:34.392534 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:34.392555 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:34.395001 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:34.398427 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:34.398438 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:34.398449 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:34.399116 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:34.399489 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:34.399496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:34.399499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:34.399657 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:34.400225 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:34.400233 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:34.400438 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 02:59:34.400707 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 02:59:34.401211 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":470,"threads_started":1}
I20260620 02:59:34.402565 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:34.380819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.0.97.61:43161)
0620 02:59:34.381641 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:34.381655 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:34.381696 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:59:34.383291 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:34.383321 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:34.383856 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:34.384316 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:34.386251 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.388447 (+ 2196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.389977 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.390660 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.390865 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:34.395706 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:34.395713 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:34.395725 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:34.395754 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:34.398666 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:34.399333 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:34.399341 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:34.399344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:34.399497 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:34.399887 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:34.399892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:34.399895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:34.400383 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:34.400669 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:34.401030 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 02:59:34.401373 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":572,"thread_start_us":375,"threads_started":1}
I20260620 02:59:34.405267 7073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:59:34 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924374, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 02:59:34.417829 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:34.405974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43161 (local address 127.0.0.1:54162)
0620 02:59:34.406154 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:34.406189 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:34.406295 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:34.406566 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:34.406569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:34.406918 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:34.407036 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:34.407043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.407969 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.407977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:34.408832 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:34.408842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.409490 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.409496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:34.409737 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:34.410326 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:34.410343 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:34.412258 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:34.415425 (+ 3167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:34.415431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:34.415435 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:34.415796 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:34.416159 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:34.416167 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:34.416171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:34.416293 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:34.416839 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:34.416843 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:34.416979 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 02:59:34.417139 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 02:59:34.417418 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":23}
I20260620 02:59:34.417835 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:34.406123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54162 (local address 127.0.97.61:43161)
0620 02:59:34.406270 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:34.406275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:34.406291 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:59:34.406395 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:34.406400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:34.406457 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:34.406551 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:34.407159 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.407849 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.408966 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.409354 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.409490 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:34.412442 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:34.412446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:34.412449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:34.412475 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:34.415284 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:34.415906 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:34.415914 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:34.415918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:34.416038 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:34.416404 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:34.416409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:34.416412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:34.416736 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:34.416875 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:34.417222 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 02:59:34.417528 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":157,"server-negotiator.queue_time_us":58}
I20260620 02:59:34.423231 7073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: Bootstrap starting.
I20260620 02:59:34.426491 7073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:34.427520 7073 log.cc:824] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:34.430457 7073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: No bootstrap required, opened a new log
I20260620 02:59:34.437630 7073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } }
I20260620 02:59:34.437985 7073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:34.438093 7073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 683840b125cc4cf791dc563ae6126dc9, State: Initialized, Role: FOLLOWER
I20260620 02:59:34.438690 7073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } }
I20260620 02:59:34.439563 7080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } } }
I20260620 02:59:34.439901 7080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:59:34.440590 7073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:59:34.449802 7091 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:59:34.449952 7091 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:59:34.452033 7006 master_runner.cc:428] Detected that this master 683840b125cc4cf791dc563ae6126dc9 is joining an existing cluster
I20260620 02:59:34.452142 7006 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:43161
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260620 02:59:34.483351 7080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:59:34.483713 7080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } }
I20260620 02:59:34.485312 7080 leader_election.cc:288] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2e5f36f1afb417f86b54cb6c0ee43a4 (127.0.97.62:35199)
I20260620 02:59:34.486274 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:34.456987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54178 (local address 127.0.97.61:43161)
0620 02:59:34.457784 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:34.457793 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:34.470161 (+ 12368us) server_negotiation.cc:407] Connection header received
0620 02:59:34.470329 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:34.470337 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:34.470412 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:34.470480 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:34.471960 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.473162 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.475337 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.476176 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.476298 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:34.479035 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:34.479044 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:34.479064 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:34.479105 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:34.483094 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:34.483987 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:34.483993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:34.483996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:34.484152 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:34.484583 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:34.484587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:34.484591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:34.484974 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:34.485137 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:34.485609 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 02:59:34.485871 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":628}
I20260620 02:59:34.486276 6555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "683840b125cc4cf791dc563ae6126dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" is_pre_election: true
I20260620 02:59:34.487038 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:34.456336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60154 (local address 127.0.97.62:35199)
0620 02:59:34.456526 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:34.456531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:34.469396 (+ 12865us) server_negotiation.cc:407] Connection header received
0620 02:59:34.469763 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:34.469769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:34.469869 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:34.469946 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:34.471555 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.472654 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.474807 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.475433 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.475532 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:34.478555 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:34.478562 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:34.478567 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:34.478608 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:34.482293 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:34.484254 (+ 1961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:34.484261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:34.484266 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:34.484507 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:34.484941 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:34.484947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:34.484950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:34.485301 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:34.485497 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:34.486111 (+ 614us) server_negotiation.cc:299] Negotiation successful
0620 02:59:34.486682 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:59:34.487254 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:34.457345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43161 (local address 127.0.0.1:54178)
0620 02:59:34.470020 (+ 12675us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:34.470041 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:34.470190 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:34.470674 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:34.470680 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:34.471467 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:34.471816 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:34.471830 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.473392 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.473399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:34.475196 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:34.475213 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.476583 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.476591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:34.476801 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:34.477510 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:34.477538 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:34.478855 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:34.483236 (+ 4381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:34.483241 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:34.483245 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:34.483859 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:34.484280 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:34.484287 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:34.484291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:34.484456 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:34.485206 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:34.485211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:34.485347 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 02:59:34.486490 (+ 1143us) client_negotiation.cc:240] Negotiation successful
0620 02:59:34.486790 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12533,"thread_start_us":11582,"threads_started":1}
I20260620 02:59:34.487435 7023 leader_election.cc:302] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 683840b125cc4cf791dc563ae6126dc9; no voters: c2e5f36f1afb417f86b54cb6c0ee43a4
I20260620 02:59:34.488142 7080 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 02:59:34.488590 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:34.456362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.0.1:60154)
0620 02:59:34.469237 (+ 12875us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:34.469259 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:34.469547 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:34.470178 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:34.470185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:34.470943 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:34.471363 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:34.471376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.472838 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.472843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:34.474638 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:34.474652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.475846 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.475853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:34.476066 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:34.477004 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:34.477037 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:34.478298 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:34.482452 (+ 4154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:34.482458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:34.482462 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:34.484104 (+ 1642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:34.484662 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:34.484668 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:34.484674 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:34.484814 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:34.485609 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:34.485613 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:34.485823 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 02:59:34.487714 (+ 1891us) client_negotiation.cc:240] Negotiation successful
0620 02:59:34.488028 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12718,"mutex_wait_us":5,"thread_start_us":13340,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Jun 20 02:59:34 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 02:59:34.515753 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:34.495536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.0.97.61:43161)
0620 02:59:34.495711 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:34.495717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:34.497498 (+ 1781us) server_negotiation.cc:407] Connection header received
0620 02:59:34.497945 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:34.497951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:34.498086 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:34.498227 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:34.500069 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.501321 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.503556 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.504291 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.504430 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:34.508888 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:34.508895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:34.508901 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:34.508942 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:34.512399 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:34.513377 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:34.513383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:34.513387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:34.513543 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:34.514060 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:34.514064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:34.514067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:34.514401 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:34.514577 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:34.515245 (+ 668us) server_negotiation.cc:299] Negotiation successful
0620 02:59:34.515442 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:59:34.515910 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:34.495166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43161 (local address 127.0.0.1:54190)
0620 02:59:34.497365 (+ 2199us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:34.497409 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:34.497674 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:34.498470 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:34.498478 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:34.499513 (+ 1035us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:34.499890 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:34.499903 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.501467 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.501475 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:34.503396 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:34.503413 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.504486 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.504495 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:34.504756 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:34.505562 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:34.505616 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:34.508666 (+ 3050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:34.512534 (+ 3868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:34.512547 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:34.512556 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:34.513214 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:34.513721 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:34.513727 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:34.513731 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:34.513931 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:34.514561 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:34.514571 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:34.514836 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 02:59:34.515073 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 02:59:34.515450 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1938,"thread_start_us":1760,"threads_started":1}
I20260620 02:59:34.518298 6545 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:43161: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "683840b125cc4cf791dc563ae6126dc9"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 43161
}
attrs {
promote: true
}
}
dest_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4"
cas_config_opid_index: -1
I20260620 02:59:34.519338 6545 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260620 02:59:34.521958 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34735
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35199,127.0.97.61:43161,127.0.97.60:34735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:59:34.528503 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 683840b125cc4cf791dc563ae6126dc9 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: true } } }
I20260620 02:59:34.529929 7048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:34.531672 7104 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:43161
I20260620 02:59:34.532105 7104 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: true } } }
I20260620 02:59:34.532299 7104 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:34.532516 7006 master.cc:537] Master@127.0.97.61:43161 shutting down...
I20260620 02:59:34.535349 7048 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 1 FOLLOWER]: Refusing update from remote peer c2e5f36f1afb417f86b54cb6c0ee43a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 02:59:34.536405 7104 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:59:34.543180 6532 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 02:59:34.544840 6532 consensus_peers.cc:487] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 -> Peer 683840b125cc4cf791dc563ae6126dc9 (127.0.97.61:43161): Couldn't send request to peer 683840b125cc4cf791dc563ae6126dc9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 02:59:34.548347 7006 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:34.548820 7006 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:34.549024 7006 tablet_replica.cc:330] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: stopping tablet replica
I20260620 02:59:34.555716 7006 master.cc:552] Master@127.0.97.61:43161 shutdown complete.
I20260620 02:59:34.555819 7006 master_runner.cc:305] Clearing existing system tablet
I20260620 02:59:34.556197 7006 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:34.556303 7006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:34.558506 7006 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 02:59:34.559892 7109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:34.560169 7006 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:34.560457 7006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 02:59:34.560539 7006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/wal
uuid: "683840b125cc4cf791dc563ae6126dc9"
format_stamp: "Formatted at 2026-06-20 02:59:34 on dist-test-slave-f1j0"
server_key: "45123c0c83027f5756e8846ef116083a"
server_key_iv: "79b2e5e4b2c070eb07efeb363b8b2ab6"
server_key_version: "encryptionkey@0"
I20260620 02:59:34.563953 7006 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:34.568459 7006 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 02:59:34.568655 7006 log.cc:1196] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 02:59:34.569015 7006 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: Deleting consensus metadata
I20260620 02:59:34.569291 7006 master_runner.cc:315] Copying system tablet from 127.0.97.62:35199
I20260620 02:59:34.571595 7006 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:35199
I20260620 02:59:34.584934 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:34.572309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.0.97.62:35199)
0620 02:59:34.572521 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:34.572530 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:34.574088 (+ 1558us) server_negotiation.cc:407] Connection header received
0620 02:59:34.574279 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:34.574284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:34.574377 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:34.574487 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:34.575297 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.576124 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.577310 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.577799 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.577929 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:34.579639 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:34.579645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:34.579650 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:34.579688 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:34.582269 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:34.582878 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:34.582884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:34.582888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:34.583011 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:34.583372 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:34.583379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:34.583382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:34.583754 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:34.583915 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:34.584277 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 02:59:34.584462 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:59:34.585043 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:34.572258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.0.1:60160)
0620 02:59:34.573966 (+ 1708us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:34.573985 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:34.574142 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:34.574509 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:34.574514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:34.574937 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:34.575150 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:34.575159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.576266 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.576271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:34.577168 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:34.577179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.577941 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.577947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:34.578155 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:34.578767 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:34.578784 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:34.579464 (+ 680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:34.582402 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:34.582409 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:34.582413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:34.582764 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:34.583122 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:34.583127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:34.583130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:34.583265 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:34.583878 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:34.583883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:34.584042 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 02:59:34.584187 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 02:59:34.584462 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1565,"thread_start_us":1489,"threads_started":1}
I20260620 02:59:34.586076 6565 tablet_copy_service.cc:140] P c2e5f36f1afb417f86b54cb6c0ee43a4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 683840b125cc4cf791dc563ae6126dc9 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60160)
I20260620 02:59:34.586308 6565 tablet_copy_service.cc:161] P c2e5f36f1afb417f86b54cb6c0ee43a4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 683840b125cc4cf791dc563ae6126dc9 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60160: session id = 683840b125cc4cf791dc563ae6126dc9-00000000000000000000000000000000
I20260620 02:59:34.589416 6565 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 02:59:34.591212 7006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 02:59:34.596402 7006 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: tablet copy: Starting download of 0 data blocks...
I20260620 02:59:34.596690 7006 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: tablet copy: Starting download of 1 WAL segments...
I20260620 02:59:34.599843 7006 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 02:59:34.603235 6565 tablet_copy_service.cc:342] P c2e5f36f1afb417f86b54cb6c0ee43a4: Request end of tablet copy session 683840b125cc4cf791dc563ae6126dc9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60160
I20260620 02:59:34.603410 6565 tablet_copy_service.cc:433] P c2e5f36f1afb417f86b54cb6c0ee43a4: ending tablet copy session 683840b125cc4cf791dc563ae6126dc9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 683840b125cc4cf791dc563ae6126dc9
I20260620 02:59:34.613322 7006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:34.616255 7116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:34.622743 7117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:34.627514 7006 server_base.cc:1034] running on GCE node
W20260620 02:59:34.627722 7119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:34.628305 7006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:34.629838 7006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:34.631140 7006 hybrid_clock.cc:648] HybridClock initialized: now 1781924374631072 us; error 32 us; skew 500 ppm
I20260620 02:59:34.631527 7006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:34 dist-test-slave-f1j0 krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924374, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:34.633989 7006 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 02:59:34.637995 7006 webserver.cc:460] Webserver started at http://127.0.97.61:37395/ using document root <none> and password file <none>
I20260620 02:59:34.638396 7006 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:34.638530 7006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:34.641678 7006 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 02:59:34.643854 7126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:34.644280 7006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260620 02:59:34.644449 7006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/wal
uuid: "683840b125cc4cf791dc563ae6126dc9"
format_stamp: "Formatted at 2026-06-20 02:59:34 on dist-test-slave-f1j0"
server_key: "45123c0c83027f5756e8846ef116083a"
server_key_iv: "79b2e5e4b2c070eb07efeb363b8b2ab6"
server_key_version: "encryptionkey@0"
I20260620 02:59:34.644686 7006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:34.662127 7006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:34.666283 7006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:34.687336 7006 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:43161
I20260620 02:59:34.687351 7177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:43161 every 8 connection(s)
I20260620 02:59:34.688516 7006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-1/data/info.pb
I20260620 02:59:34.694464 7178 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:59:34.698513 7178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: Bootstrap starting.
I20260620 02:59:34.714988 7178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:59:34.715499 7178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: Bootstrap complete.
I20260620 02:59:34.716585 7178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: true } }
I20260620 02:59:34.716884 7178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 683840b125cc4cf791dc563ae6126dc9, State: Initialized, Role: LEARNER
I20260620 02:59:34.717109 7178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: true } }
I20260620 02:59:34.718250 7178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:59:34.717974 7181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: true } } }
I20260620 02:59:34.718534 7181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:59:34.729043 7192 catalog_manager.cc:1259] Loaded cluster ID: 437db21f6b864dab8a24edfc3a3cdbeb
I20260620 02:59:34.729167 7192 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: loading cluster ID for follower catalog manager: success
I20260620 02:59:34.732669 7192 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: acquiring CA information for follower catalog manager: success
I20260620 02:59:34.735095 7192 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:59:34.883386 7001 heartbeater.cc:503] Master 127.0.97.62:35199 was elected leader, sending a full tablet report...
W20260620 02:59:34.919100 7105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:34.919435 7105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:34.919579 7105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:34.927465 7105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:59:34.927558 7105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:34.927601 7105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:34.927659 7105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:59:34.927701 7105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:59:34.937120 7105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35199,127.0.97.61:43161,127.0.97.60:34735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:34.938383 7105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:34.939597 7105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:34.951574 7197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:34.951866 7200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:34.951972 7198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:34.953019 7105 server_base.cc:1034] running on GCE node
I20260620 02:59:34.953855 7105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:34.955247 7105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:34.956468 7105 hybrid_clock.cc:648] HybridClock initialized: now 1781924374956403 us; error 53 us; skew 500 ppm
I20260620 02:59:34.956822 7105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:34 dist-test-slave-f1j0 krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924374, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:34.962744 7105 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 02:59:34.964164 7105 webserver.cc:460] Webserver started at http://127.0.97.60:45599/ using document root <none> and password file <none>
I20260620 02:59:34.964849 7105 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:34.964991 7105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:34.965286 7105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:34.968506 7105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/data/instance:
uuid: "d140ec2db4674ef1ade56ccc44494c5d"
format_stamp: "Formatted at 2026-06-20 02:59:34 on dist-test-slave-f1j0"
server_key: "7ccdcb90e35e19816dd6743cf043850b"
server_key_iv: "21bf08c470d8965beea800f71193bf9a"
server_key_version: "encryptionkey@0"
I20260620 02:59:34.969420 7105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/wal/instance:
uuid: "d140ec2db4674ef1ade56ccc44494c5d"
format_stamp: "Formatted at 2026-06-20 02:59:34 on dist-test-slave-f1j0"
server_key: "7ccdcb90e35e19816dd6743cf043850b"
server_key_iv: "21bf08c470d8965beea800f71193bf9a"
server_key_version: "encryptionkey@0"
I20260620 02:59:34.975267 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:34.968534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43161 (local address 127.0.0.1:54202)
0620 02:59:34.968799 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:34.968819 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:34.968937 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:34.969761 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:34.969766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:34.969807 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:34.969992 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:34.970007 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.971295 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.971300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:34.972492 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:34.972504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.974007 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.974017 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:34.974716 (+ 699us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:34.974774 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 02:59:34.974882 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 02:59:34.975013 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260620 02:59:34.975286 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:34.968729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.0.97.61:43161)
0620 02:59:34.969325 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:34.969329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:34.969345 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:59:34.969407 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:34.969411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:34.969476 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:34.969727 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:34.970157 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.971148 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.972647 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:34.973828 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:34.973963 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:34.974193 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:34.974933 (+ 740us) server_negotiation.cc:299] Negotiation successful
0620 02:59:34.975045 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":304,"threads_started":1}
I20260620 02:59:34.975303 7105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:59:34.976425 7153 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 02:59:34.976980 7181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2e5f36f1afb417f86b54cb6c0ee43a4. Latest consensus state: current_term: 1 leader_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: true } } }
I20260620 02:59:34.977244 7181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:59:34.979432 7208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:34.980839 7105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:59:34.981010 7105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/wal
uuid: "d140ec2db4674ef1ade56ccc44494c5d"
format_stamp: "Formatted at 2026-06-20 02:59:34 on dist-test-slave-f1j0"
server_key: "7ccdcb90e35e19816dd6743cf043850b"
server_key_iv: "21bf08c470d8965beea800f71193bf9a"
server_key_version: "encryptionkey@0"
I20260620 02:59:34.981204 7105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:35.041884 7105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:35.046556 7105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:35.046829 7105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:35.065193 7105 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34735
I20260620 02:59:35.065212 7259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34735 every 8 connection(s)
I20260620 02:59:35.066512 7105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/data/info.pb
I20260620 02:59:35.070715 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:35.074580 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 7105
I20260620 02:59:35.077348 7260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:59:35 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924374, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 02:59:35 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 02:59:35.096935 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.081345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60170 (local address 127.0.97.62:35199)
0620 02:59:35.081536 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:35.081541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:35.081941 (+ 400us) server_negotiation.cc:407] Connection header received
0620 02:59:35.082101 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:35.082107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:35.082189 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:35.082307 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:35.083725 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.084527 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.086052 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.086527 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.086669 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.090204 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:35.090208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:35.090211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:35.090241 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:35.093532 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.094380 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.094384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.094386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.094539 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.095052 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.095058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.095063 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.095433 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:35.095572 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:35.096323 (+ 751us) server_negotiation.cc:299] Negotiation successful
0620 02:59:35.096556 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:59:35.098448 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.081121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.0.1:60170)
0620 02:59:35.081789 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:35.081838 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:35.081981 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:35.082359 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:35.082363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:35.083326 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:35.083573 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.083582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.084658 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.084666 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.085911 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.085927 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.086727 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.086733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.086881 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.087446 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:35.087465 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:35.089987 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:35.093690 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.093699 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.093708 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.094284 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.094658 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.094664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.094667 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.094910 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.095563 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:35.095569 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:35.095912 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 02:59:35.096228 (+ 316us) client_negotiation.cc:240] Negotiation successful
0620 02:59:35.096748 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":3,"thread_start_us":278,"threads_started":1}
I20260620 02:59:35.098491 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.076648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51298 (local address 127.0.97.60:34735)
0620 02:59:35.077377 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:35.077387 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:35.077425 (+ 38us) server_negotiation.cc:407] Connection header received
0620 02:59:35.079223 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:35.079244 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:35.079646 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:35.079956 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:35.081577 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.083540 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.084996 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.085875 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.086050 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.090783 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:35.090791 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:35.090802 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:35.090827 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:35.094187 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.094998 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.095008 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.095011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.095172 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.095485 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.095493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.095498 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.095946 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:35.096162 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:35.096537 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 02:59:35.096874 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":531,"thread_start_us":333,"threads_started":1}
I20260620 02:59:35.100195 7260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:59:35 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924374, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 02:59:35.112952 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.101046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.0.97.61:43161)
0620 02:59:35.101404 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:35.101409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:35.101634 (+ 225us) server_negotiation.cc:407] Connection header received
0620 02:59:35.101830 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:35.101836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:35.101909 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:35.102007 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:35.102637 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.103382 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.104557 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.104990 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.105120 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.107996 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:35.108001 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:35.108004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:35.108035 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:35.110481 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.111114 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.111121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.111129 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.111254 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.111602 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.111609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.111612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.111899 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:35.112049 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:35.112427 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 02:59:35.112633 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260620 02:59:35.112947 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.101058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43161 (local address 127.0.0.1:54206)
0620 02:59:35.101520 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:35.101535 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:35.101699 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:35.102026 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:35.102029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:35.102373 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:35.102497 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.102504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.103519 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.103524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.104413 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.104423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.105145 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.105151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.105358 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.105930 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:35.105946 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:35.107801 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:35.110614 (+ 2813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.110621 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.110625 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.110992 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.111361 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.111367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.111371 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.111497 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.112012 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:35.112017 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:35.112160 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 02:59:35.112308 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 02:59:35.112605 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369}
I20260620 02:59:35.114136 7260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:59:35 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924374, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 02:59:35.126350 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.114972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.0.97.60:34735)
0620 02:59:35.115121 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:35.115126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:35.115142 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:59:35.115308 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:35.115311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:35.115358 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:35.115452 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:35.116078 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.116794 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.117906 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.118337 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.118461 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.121197 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:35.121201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:35.121204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:35.121229 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:35.123900 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.124496 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.124504 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.124508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.124635 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.124980 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.124988 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.124991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.125229 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:35.125372 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:35.125693 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 02:59:35.125999 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":58}
I20260620 02:59:35.126371 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.114822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34735 (local address 127.0.0.1:51306)
0620 02:59:35.115027 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:35.115047 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:35.115184 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:35.115478 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:35.115482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:35.115835 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:35.115951 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.115958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.116903 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.116911 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.117785 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.117795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.118463 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.118468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.118708 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.119211 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:35.119225 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:35.121022 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:35.124023 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.124030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.124034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.124387 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.124735 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.124742 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.124747 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.124883 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.125337 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:35.125342 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:35.125470 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 02:59:35.125619 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 02:59:35.125881 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":3}
I20260620 02:59:35.132417 7260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: Bootstrap starting.
I20260620 02:59:35.135637 7260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:35.136754 7260 log.cc:824] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:35.139416 7260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: No bootstrap required, opened a new log
I20260620 02:59:35.147159 7260 raft_consensus.cc:348] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } }
I20260620 02:59:35.147460 7260 raft_consensus.cc:374] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:35.147596 7260 raft_consensus.cc:729] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d140ec2db4674ef1ade56ccc44494c5d, State: Initialized, Role: FOLLOWER
I20260620 02:59:35.148159 7260 consensus_queue.cc:260] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } }
I20260620 02:59:35.148944 7266 sys_catalog.cc:455] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } } }
I20260620 02:59:35.149293 7266 sys_catalog.cc:458] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:59:35.149996 7260 sys_catalog.cc:564] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:59:35.159134 7277 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:59:35.159293 7277 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:59:35.163722 7105 master_runner.cc:428] Detected that this master d140ec2db4674ef1ade56ccc44494c5d is joining an existing cluster
I20260620 02:59:35.163843 7105 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:34735
I20260620 02:59:35.190070 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.168444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60178 (local address 127.0.97.62:35199)
0620 02:59:35.168619 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:35.168625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:35.172053 (+ 3428us) server_negotiation.cc:407] Connection header received
0620 02:59:35.172241 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:35.172250 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:35.172358 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:35.172470 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:35.174306 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.175803 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.179085 (+ 3282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.179767 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.179879 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.182657 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:35.182663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:35.182668 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:35.182706 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:35.186677 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.187600 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.187605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.187609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.187773 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.188244 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.188248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.188251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.188581 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:35.188705 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:35.189360 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 02:59:35.189629 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:59:35.191565 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.167971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.0.1:60178)
0620 02:59:35.171934 (+ 3963us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:35.171960 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:35.172116 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:35.172721 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:35.172729 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:35.173650 (+ 921us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:35.174089 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.174105 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.176061 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.176069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.178920 (+ 2851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.178932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.180164 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.180170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.180358 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.181153 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:35.181176 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:35.182436 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:35.186886 (+ 4450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.186896 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.186900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.187457 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.187921 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.187928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.187933 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.188124 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.188802 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:35.188808 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:35.189050 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 02:59:35.190661 (+ 1611us) client_negotiation.cc:240] Negotiation successful
0620 02:59:35.191018 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3820,"mutex_wait_us":2,"thread_start_us":3640,"threads_started":1}
I20260620 02:59:35.192093 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.167996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.0.97.60:34735)
0620 02:59:35.168206 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:35.168217 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:35.175436 (+ 7219us) server_negotiation.cc:407] Connection header received
0620 02:59:35.175870 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:35.175875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:35.175937 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:35.175996 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:35.177853 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.179047 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.180978 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.181776 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.181871 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.185437 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:35.185443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:35.185448 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:35.185481 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:35.189240 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.189992 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.189996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.190000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.190124 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.190495 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.190499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.190503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.190853 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:35.191004 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:35.191523 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 02:59:35.191737 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":76}
I20260620 02:59:35.194410 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.168611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34735 (local address 127.0.0.1:51320)
0620 02:59:35.175346 (+ 6735us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:35.175365 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:35.175722 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:35.176174 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:35.176179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:35.177342 (+ 1163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:35.177697 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.177711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.179242 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.179247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.180835 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.180849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.182186 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.182193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.182400 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.183070 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:35.183116 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:35.185194 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:35.189376 (+ 4182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.189381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.189385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.189878 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.190249 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.190253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.190257 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.190385 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.191070 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:35.191074 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:35.191225 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 02:59:35.193647 (+ 2422us) client_negotiation.cc:240] Negotiation successful
0620 02:59:35.193939 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6595,"thread_start_us":6809,"threads_started":1}
I20260620 02:59:35.194947 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.168009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54220 (local address 127.0.97.61:43161)
0620 02:59:35.168217 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:35.168226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:35.178765 (+ 10539us) server_negotiation.cc:407] Connection header received
0620 02:59:35.179217 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:35.179223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:35.179310 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:35.179376 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:35.180784 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.181931 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.184530 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.185259 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.185371 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.188282 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:35.188291 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:35.188295 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:35.188336 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:35.191712 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.192443 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.192451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.192455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.192587 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.193201 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.193208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.193212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.193543 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:35.193702 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:35.194397 (+ 695us) server_negotiation.cc:299] Negotiation successful
0620 02:59:35.194618 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:59:35.196190 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.168632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43161 (local address 127.0.0.1:54220)
0620 02:59:35.178651 (+ 10019us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:35.178678 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:35.179075 (+ 397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:35.179586 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:35.179591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:35.180265 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:35.180602 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.180617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.182117 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.182122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.184358 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.184375 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.185712 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.185718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.185885 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.186699 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:35.186728 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:35.188066 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:35.191874 (+ 3808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.191884 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.191887 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.192345 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.192713 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.192718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.192721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.193089 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.193769 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:35.193774 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:35.194023 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 02:59:35.195135 (+ 1112us) client_negotiation.cc:240] Negotiation successful
0620 02:59:35.195541 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9892,"mutex_wait_us":3,"spinlock_wait_cycles":24832,"thread_start_us":10136,"threads_started":1}
I20260620 02:59:35.204231 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 6606
Jun 20 02:59:35 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 02:59:35.213097 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.198248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34735 (local address 127.0.0.1:51324)
0620 02:59:35.198520 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:35.198539 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:35.198680 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:35.199098 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:35.199104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:35.199508 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:35.199645 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.199653 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.200660 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.200665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.201635 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.201648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.202501 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.202506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.202664 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.203374 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:35.203396 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:35.206465 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:35.210423 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.210428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.210432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.210907 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.211327 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.211333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.211337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.211479 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.212117 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:35.212122 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:35.212275 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 02:59:35.212436 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 02:59:35.212706 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260620 02:59:35.213516 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.198199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51324 (local address 127.0.97.60:34735)
0620 02:59:35.198375 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:35.198382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:35.198605 (+ 223us) server_negotiation.cc:407] Connection header received
0620 02:59:35.198862 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:35.198867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:35.198965 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:35.199067 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:35.199789 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.200521 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.201791 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.202344 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.202480 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.206670 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:35.206677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:35.206682 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:35.206716 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:35.210270 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.211041 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.211047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.211050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.211197 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.211602 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.211607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.211610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.211977 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:35.212133 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:35.212564 (+ 431us) server_negotiation.cc:299] Negotiation successful
0620 02:59:35.212803 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 02:59:35 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260620 02:59:35.215332 6545 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:34735: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 34735
}
attrs {
promote: true
}
}
dest_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4"
cas_config_opid_index: 5
I20260620 02:59:35.216344 6545 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: true } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: true } }
I20260620 02:59:35.217916 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d140ec2db4674ef1ade56ccc44494c5d (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: true } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: true } } }
I20260620 02:59:35.221124 7104 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:34735
I20260620 02:59:35.221608 7235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:35.221814 7105 master.cc:537] Master@127.0.97.60:34735 shutting down...
I20260620 02:59:35.225028 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.205214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.0.97.1:43727)
0620 02:59:35.205935 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:35.205942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:35.205967 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:59:35.206044 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:35.206052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:35.206341 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:35.206519 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:35.207881 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.208847 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.210139 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.210685 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.210827 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.216537 (+ 5710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:35.216552 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:35.216571 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:35.216620 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:35.220548 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.221400 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.221408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.221411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.221548 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.222064 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.222071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.222077 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.222890 (+ 813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:35.224198 (+ 1308us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:35.224355 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 02:59:35.224606 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":340,"threads_started":1}
I20260620 02:59:35.225523 7235 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [term 1 FOLLOWER]: Refusing update from remote peer c2e5f36f1afb417f86b54cb6c0ee43a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:59:35.226629 6729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:35.228767 7153 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 1 LEARNER]: Refusing update from remote peer c2e5f36f1afb417f86b54cb6c0ee43a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:59:35.231176 7290 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.003s
I20260620 02:59:35.235988 7291 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:59:35.248435 7287 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: true } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: true } } }
I20260620 02:59:35.248997 7287 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:35.256951 7153 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d140ec2db4674ef1ade56ccc44494c5d (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: true } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: true } } }
W20260620 02:59:35.259838 6529 consensus_peers.cc:487] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 -> Peer d140ec2db4674ef1ade56ccc44494c5d (127.0.97.60:34735): Couldn't send request to peer d140ec2db4674ef1ade56ccc44494c5d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 02:59:35.261323 7181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: true } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: true } } }
I20260620 02:59:35.261620 7181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:59:35.262069 7287 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4: attempting to promote NON_VOTER 683840b125cc4cf791dc563ae6126dc9 to VOTER
I20260620 02:59:35.262801 7287 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: true } }
I20260620 02:59:35.264767 7153 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 1 LEARNER]: Refusing update from remote peer c2e5f36f1afb417f86b54cb6c0ee43a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 02:59:35.265268 7287 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260620 02:59:35.265978 6529 consensus_peers.cc:487] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 -> Peer d140ec2db4674ef1ade56ccc44494c5d (127.0.97.60:34735): Couldn't send request to peer d140ec2db4674ef1ade56ccc44494c5d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 02:59:35.268042 7105 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:35.267761 7290 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 683840b125cc4cf791dc563ae6126dc9 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: true } } }
I20260620 02:59:35.268482 7105 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:35.268663 7105 tablet_replica.cc:330] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: stopping tablet replica
I20260620 02:59:35.268499 7153 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 683840b125cc4cf791dc563ae6126dc9 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: true } } }
I20260620 02:59:35.272001 7104 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: true } } }
I20260620 02:59:35.272277 7104 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:35.271961 7287 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: true } } }
I20260620 02:59:35.272617 7181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: true } } }
I20260620 02:59:35.272843 7181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:59:35.273202 7287 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:35.275604 7105 master.cc:552] Master@127.0.97.60:34735 shutdown complete.
I20260620 02:59:35.275686 7105 master_runner.cc:305] Clearing existing system tablet
I20260620 02:59:35.276062 7105 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:35.276173 7105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:35.279452 7105 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 02:59:35.281039 7296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:35.281418 7105 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:35.281695 7105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260620 02:59:35.281795 7105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/wal
uuid: "d140ec2db4674ef1ade56ccc44494c5d"
format_stamp: "Formatted at 2026-06-20 02:59:34 on dist-test-slave-f1j0"
server_key: "7ccdcb90e35e19816dd6743cf043850b"
server_key_iv: "21bf08c470d8965beea800f71193bf9a"
server_key_version: "encryptionkey@0"
I20260620 02:59:35.286655 7105 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:35.293337 7105 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 02:59:35.293478 7105 log.cc:1196] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 02:59:35.294029 7105 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: Deleting consensus metadata
I20260620 02:59:35.294310 7105 master_runner.cc:315] Copying system tablet from 127.0.97.62:35199
I20260620 02:59:35.299189 7105 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:35199
I20260620 02:59:35.314834 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.300006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60188 (local address 127.0.97.62:35199)
0620 02:59:35.300237 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:35.300244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:35.300720 (+ 476us) server_negotiation.cc:407] Connection header received
0620 02:59:35.300907 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:35.300913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:35.300993 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:35.301115 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:35.302100 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.303625 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.305419 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.306075 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.306193 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.308457 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:35.308463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:35.308469 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:35.308513 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:35.311952 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.312706 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.312712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.312716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.312870 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.313283 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.313287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.313291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.313660 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:35.313810 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:35.314266 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 02:59:35.314483 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 02:59:35.314903 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.299988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.0.1:60188)
0620 02:59:35.300595 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:35.300619 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:35.300767 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:35.301152 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:35.301156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:35.301685 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:35.301946 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.301958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.303775 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.303781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.305273 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.305286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.306232 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.306239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.306432 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.307147 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:35.307176 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:35.308274 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:35.312088 (+ 3814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.312094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.312098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.312586 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.312984 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.312989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.312993 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.313172 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.313778 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:35.313783 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:35.313968 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 02:59:35.314150 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 02:59:35.314468 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":378,"threads_started":1}
I20260620 02:59:35.315779 6565 tablet_copy_service.cc:140] P c2e5f36f1afb417f86b54cb6c0ee43a4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d140ec2db4674ef1ade56ccc44494c5d ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:60188)
I20260620 02:59:35.315972 6565 tablet_copy_service.cc:161] P c2e5f36f1afb417f86b54cb6c0ee43a4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d140ec2db4674ef1ade56ccc44494c5d at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:60188: session id = d140ec2db4674ef1ade56ccc44494c5d-00000000000000000000000000000000
I20260620 02:59:35.319650 6565 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 02:59:35.321533 7105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 02:59:35.328469 7105 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: tablet copy: Starting download of 0 data blocks...
I20260620 02:59:35.328751 7105 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: tablet copy: Starting download of 1 WAL segments...
I20260620 02:59:35.331060 7105 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 02:59:35.335871 6565 tablet_copy_service.cc:342] P c2e5f36f1afb417f86b54cb6c0ee43a4: Request end of tablet copy session d140ec2db4674ef1ade56ccc44494c5d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:60188
I20260620 02:59:35.336004 6565 tablet_copy_service.cc:433] P c2e5f36f1afb417f86b54cb6c0ee43a4: ending tablet copy session d140ec2db4674ef1ade56ccc44494c5d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d140ec2db4674ef1ade56ccc44494c5d
I20260620 02:59:35.348320 7105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:35.354838 7303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:35.355899 7304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:35.362993 7306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:35.363595 7105 server_base.cc:1034] running on GCE node
I20260620 02:59:35.364045 7105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:35.364655 7105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:35.365839 7105 hybrid_clock.cc:648] HybridClock initialized: now 1781924375365788 us; error 42 us; skew 500 ppm
I20260620 02:59:35.366070 7105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:35 dist-test-slave-f1j0 krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924375, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:35.368229 7105 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 02:59:35.374620 7105 webserver.cc:460] Webserver started at http://127.0.97.60:33195/ using document root <none> and password file <none>
I20260620 02:59:35.375021 7105 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:35.375159 7105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:35.378301 7105 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 02:59:35.380281 7313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:35.380947 7105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 02:59:35.381175 7105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/wal
uuid: "d140ec2db4674ef1ade56ccc44494c5d"
format_stamp: "Formatted at 2026-06-20 02:59:34 on dist-test-slave-f1j0"
server_key: "7ccdcb90e35e19816dd6743cf043850b"
server_key_iv: "21bf08c470d8965beea800f71193bf9a"
server_key_version: "encryptionkey@0"
I20260620 02:59:35.381484 7105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:35.383764 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 6606
I20260620 02:59:35.395846 7105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:35.400643 7105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:35.408196 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43727
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35199
--webserver_port=34107
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:35199,127.0.97.61:43161
--tserver_master_addrs=127.0.97.62:35199,127.0.97.61:43161,127.0.97.60:34735 with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:59:35.421427 7105 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34735
I20260620 02:59:35.421479 7365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34735 every 8 connection(s)
I20260620 02:59:35.422752 7105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/master-2/data/info.pb
I20260620 02:59:35.428226 7366 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:59:35.432094 7366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: Bootstrap starting.
I20260620 02:59:35.450696 7366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:59:35.451221 7366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: Bootstrap complete.
I20260620 02:59:35.452338 7366 raft_consensus.cc:348] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: true } }
I20260620 02:59:35.452636 7366 raft_consensus.cc:729] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d140ec2db4674ef1ade56ccc44494c5d, State: Initialized, Role: LEARNER
I20260620 02:59:35.452904 7366 consensus_queue.cc:260] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: true } }
I20260620 02:59:35.453982 7366 sys_catalog.cc:564] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:59:35.453742 7369 sys_catalog.cc:455] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: true } } }
I20260620 02:59:35.454252 7369 sys_catalog.cc:458] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [sys.catalog]: This master's current role is: LEARNER
I20260620 02:59:35.467002 7380 catalog_manager.cc:1259] Loaded cluster ID: 437db21f6b864dab8a24edfc3a3cdbeb
I20260620 02:59:35.467134 7380 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: loading cluster ID for follower catalog manager: success
I20260620 02:59:35.471099 7380 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: acquiring CA information for follower catalog manager: success
I20260620 02:59:35.473402 7380 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:59:35.726588 7382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.719255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51338 (local address 127.0.97.60:34735)
0620 02:59:35.720128 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:35.720133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:35.720151 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:59:35.720209 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:35.720212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:35.720276 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:35.720377 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:35.720824 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.721978 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.724788 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.725557 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.725724 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.725883 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:35.726237 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 02:59:35.726341 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":639,"threads_started":1}
I20260620 02:59:35.726715 7381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.718922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34735 (local address 127.0.0.1:51338)
0620 02:59:35.719836 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:35.719859 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:35.720014 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:35.720437 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:35.720442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:35.720476 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:35.720683 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.720695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.722127 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.722132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.724609 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.724630 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.725813 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.725820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.726044 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.726090 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 02:59:35.726205 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 02:59:35.726376 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":595,"threads_started":1}
I20260620 02:59:35.727576 7340 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 02:59:35.728230 7369 sys_catalog.cc:455] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2e5f36f1afb417f86b54cb6c0ee43a4. Latest consensus state: current_term: 1 leader_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: true } } }
I20260620 02:59:35.728550 7369 sys_catalog.cc:458] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [sys.catalog]: This master's current role is: LEARNER
W20260620 02:59:35.758817 7351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:35.759137 7351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:35.759310 7351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:35.766917 7351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:35.767066 7351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:35.767249 7351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:59:35.777697 7351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34107
--tserver_master_addrs=127.0.97.62:35199,127.0.97.61:43161,127.0.97.60:34735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:35.779076 7351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:35.780439 7351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:35.796144 7390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:35.796208 7388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:35.796231 7387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:35.796914 7351 server_base.cc:1034] running on GCE node
I20260620 02:59:35.797672 7351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:35.798748 7351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:35.799973 7351 hybrid_clock.cc:648] HybridClock initialized: now 1781924375799940 us; error 28 us; skew 500 ppm
I20260620 02:59:35.800316 7351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:35 dist-test-slave-f1j0 krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924375, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:35.803548 7351 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 02:59:35.805469 7351 webserver.cc:460] Webserver started at http://127.0.97.1:34107/ using document root <none> and password file <none>
I20260620 02:59:35.806201 7351 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:35.806362 7351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:35.811709 7351 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 02:59:35.814733 7397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:35.816303 7351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:59:35.816468 7351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "3dc18659e29e43eea11d6a12dd573c8d"
format_stamp: "Formatted at 2026-06-20 02:59:32 on dist-test-slave-f1j0"
server_key: "416e0fcfd0cbc0036a22edda7e5bd6b1"
server_key_iv: "7e290780d741c2649f7bca95c59cf0eb"
server_key_version: "encryptionkey@0"
I20260620 02:59:35.817946 7351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:35.844786 7351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:35.849391 7351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:35.849653 7351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:35.850622 7351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:35.852002 7351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:35.852085 7351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:35.852232 7351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:35.852329 7351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:35.886063 7351 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43727
I20260620 02:59:35.886209 7509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43727 every 8 connection(s)
I20260620 02:59:35.887693 7351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:59:35.897598 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 7351
I20260620 02:59:35.897990 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 6740
Jun 20 02:59:35 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924375, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 02:59:35 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924375, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 02:59:35.925163 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.890492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40543 (local address 127.0.97.62:35199)
0620 02:59:35.891639 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:35.891645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:35.900976 (+ 9331us) server_negotiation.cc:407] Connection header received
0620 02:59:35.901064 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:35.901072 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:35.901213 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:35.901338 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:35.904285 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.905820 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.909118 (+ 3298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.910037 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.910173 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.916832 (+ 6659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:35.916842 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:35.916848 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:35.916889 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:35.920867 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.921975 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.921981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.921984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.922180 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.922860 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.922865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.922868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.923280 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:35.923403 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:35.924471 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0620 02:59:35.924699 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1037,"thread_start_us":994,"threads_started":1}
I20260620 02:59:35.928143 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.891720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.1:40543)
0620 02:59:35.896831 (+ 5111us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:35.896899 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:35.900437 (+ 3538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:35.901950 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:35.901968 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:35.903043 (+ 1075us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:35.903999 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.904034 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.906068 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.906077 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.908899 (+ 2822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.908933 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.910611 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.910621 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.911056 (+ 435us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.912061 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:35.912101 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:35.916565 (+ 4464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:35.921044 (+ 4479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.921052 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.921072 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.921822 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.922327 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.922332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.922335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.922683 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.923515 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:35.923523 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:35.924001 (+ 478us) client_negotiation.cc:769] Sending connection context
0620 02:59:35.925412 (+ 1411us) client_negotiation.cc:240] Negotiation successful
0620 02:59:35.926427 (+ 1015us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4699,"thread_start_us":4444,"threads_started":1}
Jun 20 02:59:35 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924375, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 02:59:35.929939 7511 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35199
I20260620 02:59:35.930274 7511 heartbeater.cc:464] Registering TS with master...
Jun 20 02:59:35 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260620 02:59:35.931263 7511 heartbeater.cc:511] Master 127.0.97.62:35199 requested a full tablet report, sending...
I20260620 02:59:35.933187 6545 ts_manager.cc:194] Re-registered known tserver with Master: 3dc18659e29e43eea11d6a12dd573c8d (127.0.97.1:43727)
I20260620 02:59:35.934281 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40543
I20260620 02:59:35.934648 7382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.891298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40361 (local address 127.0.97.60:34735)
0620 02:59:35.891483 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:35.891489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:35.913031 (+ 21542us) server_negotiation.cc:407] Connection header received
0620 02:59:35.914055 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:35.914060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:35.914139 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:35.914208 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:35.916536 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.917678 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.919462 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.920140 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.920262 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.925362 (+ 5100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:35.925370 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:35.925375 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:35.925413 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:35.929094 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.932516 (+ 3422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.932521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.932525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.932678 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.933094 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.933098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.933101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.933445 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:35.933601 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:35.934091 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 02:59:35.934339 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 02:59:35.935592 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.892028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34735 (local address 127.0.97.1:40361)
0620 02:59:35.912886 (+ 20858us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:35.912912 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:35.913898 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:35.914437 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:35.914442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:35.915999 (+ 1557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:35.916379 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.916394 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.917850 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.917857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.918991 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.919002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.920809 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.920815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.921005 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.922267 (+ 1262us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:35.922308 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:35.925149 (+ 2841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:35.929265 (+ 4116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.929270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.929274 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.929861 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.932830 (+ 2969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.932835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.932838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.932973 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.933678 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:35.933683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:35.933830 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 02:59:35.934892 (+ 1062us) client_negotiation.cc:240] Negotiation successful
0620 02:59:35.935185 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20723,"thread_start_us":20557,"threads_started":1}
I20260620 02:59:35.938066 7510 heartbeater.cc:347] Connected to a master server at 127.0.97.60:34735
I20260620 02:59:35.938208 7510 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:35.938604 7510 heartbeater.cc:511] Master 127.0.97.60:34735 requested a full tablet report, sending...
I20260620 02:59:35.938815 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.899169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59288 (local address 127.0.97.2:44039)
0620 02:59:35.912191 (+ 13022us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:35.912203 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:35.913645 (+ 1442us) server_negotiation.cc:407] Connection header received
0620 02:59:35.913855 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:35.913864 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:35.914162 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:35.914309 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:35.917342 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.920857 (+ 3515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.922821 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.923758 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.923889 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.931229 (+ 7340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:35.931248 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:35.931261 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:35.931295 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:35.935521 (+ 4226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.936462 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.936474 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.936480 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.936635 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.937081 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.937086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.937088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.937482 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:35.937824 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:35.938167 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 02:59:35.938438 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12805,"thread_start_us":13024,"threads_started":1}
I20260620 02:59:35.938886 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.891913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51315 (local address 127.0.97.61:43161)
0620 02:59:35.920803 (+ 28890us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:35.920810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:35.920837 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:59:35.920893 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:35.920898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:35.920989 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:35.921093 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:35.922708 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.923893 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.925951 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.926474 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.931358 (+ 4884us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.931612 (+ 254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:35.931617 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:35.931621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:35.931666 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:35.935647 (+ 3981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.936590 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.936595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.936598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.936738 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:35.937125 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:35.937130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:35.937133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:35.937387 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:35.937543 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:35.938384 (+ 841us) server_negotiation.cc:299] Negotiation successful
0620 02:59:35.938601 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28778,"thread_start_us":29294,"threads_started":1}
I20260620 02:59:35.939728 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:35.892651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43161 (local address 127.0.97.1:51315)
0620 02:59:35.916024 (+ 23373us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:35.916043 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:35.916165 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:35.921437 (+ 5272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:35.921442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:35.922137 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:35.922503 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.922516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.924072 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.924078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.925745 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:35.925759 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:35.926695 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:35.926702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:35.926899 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:35.927682 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:35.927706 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:35.931182 (+ 3476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:35.935963 (+ 4781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.935968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.935972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.936444 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.936843 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:35.936851 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:35.936856 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:35.937025 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:35.937508 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:35.937513 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:35.937689 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 02:59:35.937824 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 02:59:35.938134 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23252,"thread_start_us":22509,"threads_started":1}
I20260620 02:59:35.940169 6863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:35.940363 7330 ts_manager.cc:194] Registered new tserver with Master: 3dc18659e29e43eea11d6a12dd573c8d (127.0.97.1:43727)
I20260620 02:59:35.940589 7512 heartbeater.cc:347] Connected to a master server at 127.0.97.61:43161
I20260620 02:59:35.940738 7512 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:35.941154 7512 heartbeater.cc:511] Master 127.0.97.61:43161 requested a full tablet report, sending...
I20260620 02:59:35.943255 7143 ts_manager.cc:194] Registered new tserver with Master: 3dc18659e29e43eea11d6a12dd573c8d (127.0.97.1:43727)
I20260620 02:59:36.024777 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 6740
I20260620 02:59:36.041260 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44039
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35199
--webserver_port=45041
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:35199,127.0.97.61:43161
--tserver_master_addrs=127.0.97.62:35199,127.0.97.61:43161,127.0.97.60:34735 with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:59:36.285118 7287 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: true } } }
I20260620 02:59:36.285595 7287 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:36.286952 7287 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4: attempting to promote NON_VOTER d140ec2db4674ef1ade56ccc44494c5d to VOTER
I20260620 02:59:36.287520 7287 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: false } }
I20260620 02:59:36.289763 7153 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 1 FOLLOWER]: Refusing update from remote peer c2e5f36f1afb417f86b54cb6c0ee43a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 02:59:36.289798 7340 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [term 1 LEARNER]: Refusing update from remote peer c2e5f36f1afb417f86b54cb6c0ee43a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 02:59:36.290393 7287 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 02:59:36.290761 7524 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 02:59:36.295071 7523 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d140ec2db4674ef1ade56ccc44494c5d (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: false } } }
I20260620 02:59:36.295644 7153 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d140ec2db4674ef1ade56ccc44494c5d (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: false } } }
I20260620 02:59:36.295806 7340 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d140ec2db4674ef1ade56ccc44494c5d (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: false } } }
I20260620 02:59:36.299175 7523 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: false } } }
I20260620 02:59:36.299306 7524 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: false } } }
I20260620 02:59:36.299521 7523 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:36.299556 7524 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:36.299881 7524 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: false } } }
I20260620 02:59:36.300100 7524 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:36.301016 7530 sys_catalog.cc:455] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: false } } }
I20260620 02:59:36.301465 7530 sys_catalog.cc:458] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:59:36.304850 7529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: false } } }
I20260620 02:59:36.305244 7529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 02:59:36.379945 7522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:36.380246 7522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:36.380349 7522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:36.387665 7522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:36.387776 7522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:36.387887 7522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:59:36.397921 7522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45041
--tserver_master_addrs=127.0.97.62:35199,127.0.97.61:43161,127.0.97.60:34735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:36.399250 7522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:36.400557 7522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:36.413932 7535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:36.413856 7536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:36.413901 7538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:36.414780 7522 server_base.cc:1034] running on GCE node
I20260620 02:59:36.415391 7522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:36.416489 7522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:36.417706 7522 hybrid_clock.cc:648] HybridClock initialized: now 1781924376417645 us; error 35 us; skew 500 ppm
I20260620 02:59:36.418094 7522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:36 dist-test-slave-f1j0 krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924376, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:36.421345 7522 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 02:59:36.423061 7522 webserver.cc:460] Webserver started at http://127.0.97.2:45041/ using document root <none> and password file <none>
I20260620 02:59:36.423784 7522 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:36.423928 7522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:36.429543 7522 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 02:59:36.432482 7545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:36.434016 7522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:59:36.434207 7522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "0fe8419e2d0a4217880d7b0ca459c073"
format_stamp: "Formatted at 2026-06-20 02:59:33 on dist-test-slave-f1j0"
server_key: "8f5310ac57179c6fc395f6218cf443a1"
server_key_iv: "246844326046e5e598b68d36801d0f2e"
server_key_version: "encryptionkey@0"
I20260620 02:59:36.435210 7522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:36.448333 7522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:36.452121 7522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:36.452396 7522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:36.453305 7522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:36.454869 7522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:36.454984 7522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:36.455121 7522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:36.455209 7522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:36.487360 7522 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44039
I20260620 02:59:36.487445 7657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44039 every 8 connection(s)
I20260620 02:59:36.488853 7522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:59:36.492985 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 7522
I20260620 02:59:36.493306 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 6873
Jun 20 02:59:36 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260620 02:59:36.531944 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:36.495026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41038 (local address 127.0.97.3:44065)
0620 02:59:36.502865 (+ 7839us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:36.502879 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:36.504236 (+ 1357us) server_negotiation.cc:407] Connection header received
0620 02:59:36.504341 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:36.504354 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:36.504710 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:36.504890 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:36.509203 (+ 4313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:36.511067 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:36.513369 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:36.514464 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:36.514593 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:36.522776 (+ 8183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:36.522817 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:36.522843 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:36.522908 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:36.527616 (+ 4708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:36.528774 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:36.528784 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:36.528787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:36.528960 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:36.529481 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:36.529489 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:36.529495 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:36.530041 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:36.530349 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:36.531052 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 02:59:36.531377 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7103,"thread_start_us":6943,"threads_started":1}
I20260620 02:59:36.534963 6996 generic_service.cc:196] Checking for leaks (request via RPC)
Jun 20 02:59:36 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924376, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 02:59:36 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924376, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 02:59:36 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924376, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 02:59:36.557832 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:36.494481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36653 (local address 127.0.97.60:34735)
0620 02:59:36.534136 (+ 39655us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:36.534143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:36.536698 (+ 2555us) server_negotiation.cc:407] Connection header received
0620 02:59:36.536761 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:36.536767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:36.536845 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:36.536953 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:36.539717 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:36.540855 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:36.543627 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:36.544293 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:36.544386 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:36.550244 (+ 5858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:36.550251 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:36.550257 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:36.550296 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:36.553803 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:36.554834 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:36.554840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:36.554845 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:36.554990 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:36.555536 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:36.555542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:36.555546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:36.555964 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:36.556106 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:36.557194 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0620 02:59:36.557445 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39485,"thread_start_us":39326,"threads_started":1}
I20260620 02:59:36.559245 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:36.494170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42755 (local address 127.0.97.61:43161)
0620 02:59:36.535376 (+ 41206us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:36.535385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:36.537429 (+ 2044us) server_negotiation.cc:407] Connection header received
0620 02:59:36.537634 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:36.537642 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:36.537737 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:36.537845 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:36.541334 (+ 3489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:36.542462 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:36.544349 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:36.544987 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:36.545123 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:36.551530 (+ 6407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:36.551538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:36.551546 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:36.551597 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:36.555053 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:36.555892 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:36.555898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:36.555901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:36.556044 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:36.556473 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:36.556480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:36.556484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:36.556866 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:36.557036 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:36.558685 (+ 1649us) server_negotiation.cc:299] Negotiation successful
0620 02:59:36.558923 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41034,"thread_start_us":40738,"threads_started":1}
I20260620 02:59:36.560050 7669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:36.492452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34735 (local address 127.0.97.2:36653)
0620 02:59:36.536382 (+ 43930us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:36.536406 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:36.536572 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:36.537012 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:36.537020 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:36.538466 (+ 1446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:36.539527 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:36.539542 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:36.541050 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:36.541060 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:36.543476 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:36.543489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:36.544683 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:36.544690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:36.544990 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:36.546072 (+ 1082us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:36.546104 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:36.549968 (+ 3864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:36.553972 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:36.553984 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:36.554004 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:36.554692 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:36.555135 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:36.555139 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:36.555142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:36.555397 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:36.556184 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:36.556192 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:36.556717 (+ 525us) client_negotiation.cc:769] Sending connection context
0620 02:59:36.558100 (+ 1383us) client_negotiation.cc:240] Negotiation successful
0620 02:59:36.558520 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43732,"mutex_wait_us":5842,"spinlock_wait_cycles":38016,"thread_start_us":37773,"threads_started":1}
I20260620 02:59:36.560170 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:36.495252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43161 (local address 127.0.97.2:42755)
0620 02:59:36.537320 (+ 42068us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:36.537344 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:36.537480 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:36.538035 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:36.538039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:36.538696 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:36.541103 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:36.541122 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:36.542621 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:36.542626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:36.544186 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:36.544198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:36.545993 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:36.546005 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:36.546220 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:36.546884 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:36.546906 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:36.551294 (+ 4388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:36.555187 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:36.555192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:36.555198 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:36.555721 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:36.556168 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:36.556173 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:36.556176 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:36.556351 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:36.557119 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:36.557124 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:36.557247 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 02:59:36.557408 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 02:59:36.558005 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40868,"mutex_wait_us":25,"thread_start_us":39027,"threads_started":1}
I20260620 02:59:36.560609 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:36.491821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34561 (local address 127.0.97.62:35199)
0620 02:59:36.536174 (+ 44353us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:36.536183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:36.536207 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:59:36.536285 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:36.536293 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:36.536396 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:36.536522 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:36.539728 (+ 3206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:36.540853 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:36.543667 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:36.544352 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:36.544486 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:36.550775 (+ 6289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:36.550783 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:36.550790 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:36.550862 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:36.555370 (+ 4508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:36.556231 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:36.556236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:36.556240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:36.556451 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:36.556882 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:36.556929 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:36.556934 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:36.557331 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:36.559223 (+ 1892us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:36.559601 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 02:59:36.559827 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44208,"thread_start_us":45326,"threads_started":1}
I20260620 02:59:36.561335 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:36.492452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.2:34561)
0620 02:59:36.504030 (+ 11578us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:36.504107 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:36.507655 (+ 3548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:36.536864 (+ 29209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:36.536882 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:36.538070 (+ 1188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:36.539513 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:36.539542 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:36.541031 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:36.541037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:36.543480 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:36.543496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:36.544752 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:36.544758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:36.544990 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:36.546003 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:36.546039 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:36.550492 (+ 4453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:36.555516 (+ 5024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:36.555521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:36.555525 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:36.556076 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:36.556585 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:36.556590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:36.556594 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:36.556755 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:36.557487 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:36.557492 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:36.557670 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 02:59:36.557787 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 02:59:36.558363 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11150,"mutex_wait_us":227,"spinlock_wait_cycles":27008,"thread_start_us":15605,"threads_started":1}
I20260620 02:59:36.561653 7658 heartbeater.cc:347] Connected to a master server at 127.0.97.60:34735
I20260620 02:59:36.562758 7658 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:36.563279 7659 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35199
I20260620 02:59:36.563409 7659 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:36.563743 7658 heartbeater.cc:511] Master 127.0.97.60:34735 requested a full tablet report, sending...
I20260620 02:59:36.563853 7659 heartbeater.cc:511] Master 127.0.97.62:35199 requested a full tablet report, sending...
I20260620 02:59:36.561894 7660 heartbeater.cc:347] Connected to a master server at 127.0.97.61:43161
I20260620 02:59:36.564617 7660 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:36.565065 7660 heartbeater.cc:511] Master 127.0.97.61:43161 requested a full tablet report, sending...
I20260620 02:59:36.565382 6545 ts_manager.cc:194] Re-registered known tserver with Master: 0fe8419e2d0a4217880d7b0ca459c073 (127.0.97.2:44039)
I20260620 02:59:36.566105 7330 ts_manager.cc:194] Registered new tserver with Master: 0fe8419e2d0a4217880d7b0ca459c073 (127.0.97.2:44039)
I20260620 02:59:36.566183 7143 ts_manager.cc:194] Registered new tserver with Master: 0fe8419e2d0a4217880d7b0ca459c073 (127.0.97.2:44039)
I20260620 02:59:36.566511 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34561
I20260620 02:59:36.629921 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 6873
I20260620 02:59:36.646262 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44065
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35199
--webserver_port=38173
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:35199,127.0.97.61:43161
--tserver_master_addrs=127.0.97.62:35199,127.0.97.61:43161,127.0.97.60:34735 with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:59:36.937764 7511 heartbeater.cc:503] Master 127.0.97.62:35199 was elected leader, sending a full tablet report...
W20260620 02:59:36.982755 7671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:36.983104 7671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:36.983202 7671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:36.990447 7671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:36.990562 7671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:36.990676 7671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:59:37.001088 7671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38173
--tserver_master_addrs=127.0.97.62:35199,127.0.97.61:43161,127.0.97.60:34735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:37.002509 7671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:37.003794 7671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:37.017010 7676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:37.016994 7677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:37.017262 7679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:37.017850 7671 server_base.cc:1034] running on GCE node
I20260620 02:59:37.018445 7671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:37.019375 7671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:37.020613 7671 hybrid_clock.cc:648] HybridClock initialized: now 1781924377020528 us; error 81 us; skew 500 ppm
I20260620 02:59:37.020922 7671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:37 dist-test-slave-f1j0 krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924377, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:37.024288 7671 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 02:59:37.025823 7671 webserver.cc:460] Webserver started at http://127.0.97.3:38173/ using document root <none> and password file <none>
I20260620 02:59:37.026499 7671 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:37.026645 7671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:37.031841 7671 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 02:59:37.034618 7686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:37.035982 7671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:59:37.036157 7671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "f27e01b9b2cc4d48a7caddbdce674145"
format_stamp: "Formatted at 2026-06-20 02:59:33 on dist-test-slave-f1j0"
server_key: "7a56acfdd3dd51f80463cb6b8a7d6783"
server_key_iv: "61523f33cfbd436ae2494f7f9ad2165b"
server_key_version: "encryptionkey@0"
I20260620 02:59:37.037140 7671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:37.053148 7671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:37.057166 7671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:37.057458 7671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:37.058480 7671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:37.059763 7671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:37.059846 7671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:37.059990 7671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:37.060082 7671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:37.093299 7671 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44065
I20260620 02:59:37.093353 7798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44065 every 8 connection(s)
I20260620 02:59:37.094787 7671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:59:37.096666 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 7671
Jun 20 02:59:37 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924377, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 02:59:37 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924377, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 02:59:37 dist-test-slave-f1j0 krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924377, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 02:59:37.163934 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.121956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60196 (local address 127.0.97.62:35199)
0620 02:59:37.142595 (+ 20639us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:37.142602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:37.142626 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:59:37.142684 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:37.142689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:37.142775 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:37.142884 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:37.146220 (+ 3336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.147424 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.150249 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.151641 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.151766 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.155534 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:37.155541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:37.155545 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:37.155585 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:37.160087 (+ 4502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.160928 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.160933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.160936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.161115 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.161551 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.161557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.161562 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.161976 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:37.162222 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:37.163277 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0620 02:59:37.163506 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20464,"thread_start_us":20348,"threads_started":1}
I20260620 02:59:37.168735 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.103366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34735 (local address 127.0.97.3:54025)
0620 02:59:37.120461 (+ 17095us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:37.120476 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:37.122241 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:37.136432 (+ 14191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:37.136439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:37.143150 (+ 6711us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:37.143903 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:37.143922 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.145892 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.145897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:37.147452 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:37.147468 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.148527 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.148534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:37.148740 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.149417 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:37.149439 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:37.152794 (+ 3355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:37.159148 (+ 6354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:37.159158 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:37.159178 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:37.159912 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:37.163780 (+ 3868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:37.163785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:37.163788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:37.164617 (+ 829us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:37.165317 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:37.165328 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:37.165840 (+ 512us) client_negotiation.cc:769] Sending connection context
0620 02:59:37.166160 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 02:59:37.167052 (+ 892us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17000,"thread_start_us":13956,"threads_started":1}
I20260620 02:59:37.169492 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.097832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35285 (local address 127.0.97.61:43161)
0620 02:59:37.113121 (+ 15289us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:37.113131 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:37.118953 (+ 5822us) server_negotiation.cc:407] Connection header received
0620 02:59:37.122033 (+ 3080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:37.122041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:37.122153 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:37.122269 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:37.126376 (+ 4107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.127882 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.131353 (+ 3471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.132263 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.132394 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.160887 (+ 28493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:37.160895 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:37.160903 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:37.160953 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:37.165613 (+ 4660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.166408 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.166414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.166417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.166584 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.167041 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.167046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.167051 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.167406 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:37.167568 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:37.168904 (+ 1336us) server_negotiation.cc:299] Negotiation successful
0620 02:59:37.169133 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15114,"thread_start_us":15302,"threads_started":1}
I20260620 02:59:37.170008 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.134677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54230 (local address 127.0.97.61:43161)
0620 02:59:37.149703 (+ 15026us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:37.149712 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:37.150759 (+ 1047us) server_negotiation.cc:407] Connection header received
0620 02:59:37.151264 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:37.151271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:37.151362 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:37.151461 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:37.154346 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.155538 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.157531 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.158227 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.158347 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.162257 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:37.162265 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:37.162272 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:37.162308 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:37.166010 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.166883 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.166889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.166892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.167076 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.167547 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.167554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.167560 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.167886 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:37.168147 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:37.169391 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0620 02:59:37.169643 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14811,"thread_start_us":16303,"threads_started":1}
I20260620 02:59:37.170845 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.103878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54025 (local address 127.0.97.60:34735)
0620 02:59:37.116620 (+ 12742us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:37.116629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:37.120588 (+ 3959us) server_negotiation.cc:407] Connection header received
0620 02:59:37.136161 (+ 15573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:37.136169 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:37.136270 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:37.136392 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:37.144119 (+ 7727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.145716 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.147669 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.148309 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.150848 (+ 2539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.153029 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:37.153041 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:37.153048 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:37.153090 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:37.158713 (+ 5623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.163504 (+ 4791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.163509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.163512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.163651 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.164780 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.164785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.164789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.165140 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:37.170051 (+ 4911us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:37.170455 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 02:59:37.170617 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12586,"thread_start_us":12474,"threads_started":1}
I20260620 02:59:37.171710 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.098655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43161 (local address 127.0.97.3:35285)
0620 02:59:37.118452 (+ 19797us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:37.118474 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:37.121511 (+ 3037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:37.123300 (+ 1789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:37.123318 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:37.124570 (+ 1252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:37.125800 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:37.125839 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.128223 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.128232 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:37.131081 (+ 2849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:37.131104 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.132861 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.132870 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:37.133295 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.134337 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:37.134371 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:37.160660 (+ 26289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:37.165758 (+ 5098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:37.165765 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:37.165770 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:37.166270 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:37.166715 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:37.166721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:37.166724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:37.166867 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:37.167631 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:37.167637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:37.167768 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 02:59:37.167923 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 02:59:37.168667 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18772,"spinlock_wait_cycles":4992,"thread_start_us":20906,"threads_started":1}
I20260620 02:59:37.172207 7799 heartbeater.cc:347] Connected to a master server at 127.0.97.60:34735
I20260620 02:59:37.172508 7801 heartbeater.cc:347] Connected to a master server at 127.0.97.61:43161
I20260620 02:59:37.172569 7799 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:37.172644 7801 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:37.173558 7799 heartbeater.cc:511] Master 127.0.97.60:34735 requested a full tablet report, sending...
I20260620 02:59:37.173606 7801 heartbeater.cc:511] Master 127.0.97.61:43161 requested a full tablet report, sending...
I20260620 02:59:37.174129 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.099494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.3:38789)
0620 02:59:37.118129 (+ 18635us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:37.118178 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:37.121594 (+ 3416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:37.130170 (+ 8576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:37.130178 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:37.130881 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:37.131276 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:37.131292 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.134644 (+ 3352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.134653 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:37.141848 (+ 7195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:37.141871 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.143975 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.143984 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:37.144252 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.147252 (+ 3000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:37.147287 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:37.152620 (+ 5333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:37.162096 (+ 9476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:37.162101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:37.162105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:37.162615 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:37.163042 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:37.163049 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:37.163053 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:37.163327 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:37.166654 (+ 3327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:37.166659 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:37.167189 (+ 530us) client_negotiation.cc:769] Sending connection context
0620 02:59:37.173320 (+ 6131us) client_negotiation.cc:240] Negotiation successful
0620 02:59:37.173667 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18499,"mutex_wait_us":1,"spinlock_wait_cycles":49536,"thread_start_us":18307,"threads_started":1}
I20260620 02:59:37.175451 7330 ts_manager.cc:194] Registered new tserver with Master: f27e01b9b2cc4d48a7caddbdce674145 (127.0.97.3:44065)
I20260620 02:59:37.176371 7143 ts_manager.cc:194] Registered new tserver with Master: f27e01b9b2cc4d48a7caddbdce674145 (127.0.97.3:44065)
I20260620 02:59:37.169344 6545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 02:59:37.177807 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.099523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38789 (local address 127.0.97.62:35199)
0620 02:59:37.114950 (+ 15427us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:37.114959 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:37.118298 (+ 3339us) server_negotiation.cc:407] Connection header received
0620 02:59:37.129591 (+ 11293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:37.129613 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:37.129747 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:37.129852 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:37.132843 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.134366 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.142494 (+ 8128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.143420 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.143547 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.156516 (+ 12969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:37.156523 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:37.156527 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:37.156565 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:37.161027 (+ 4462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.162750 (+ 1723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.162755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.162758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.162889 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.164033 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.164039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.164042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.164388 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:37.164507 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:37.176677 (+ 12170us) server_negotiation.cc:299] Negotiation successful
0620 02:59:37.177326 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15267,"thread_start_us":15448,"threads_started":1}
I20260620 02:59:37.178597 7800 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35199
I20260620 02:59:37.178721 7800 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:37.179150 7800 heartbeater.cc:511] Master 127.0.97.62:35199 requested a full tablet report, sending...
W20260620 02:59:37.179193 6545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:59:37.180241 6544 ts_manager.cc:194] Re-registered known tserver with Master: f27e01b9b2cc4d48a7caddbdce674145 (127.0.97.3:44065)
I20260620 02:59:37.181486 6544 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38789
I20260620 02:59:37.183928 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.122627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.0.97.60:34735)
0620 02:59:37.141495 (+ 18868us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:37.141504 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:37.141531 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:59:37.141608 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:37.141616 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:37.141710 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:37.141861 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:37.152862 (+ 11001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.154222 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.157684 (+ 3462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.161795 (+ 4111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.161894 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.176524 (+ 14630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:37.176532 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:37.176536 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:37.176573 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:37.180180 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.180859 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.180868 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.180871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.180981 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.181349 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.181354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.181357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.181801 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:37.183165 (+ 1364us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:37.183289 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 02:59:37.183512 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18637,"thread_start_us":19437,"threads_started":1}
I20260620 02:59:37.218727 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.202095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44065 (local address 127.0.0.1:41054)
0620 02:59:37.204448 (+ 2353us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:37.204474 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:37.204641 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:37.209745 (+ 5104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:37.209753 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:37.209796 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:37.210312 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:37.210329 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.213360 (+ 3031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.213364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:37.215724 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:37.215738 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.217873 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.217878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:37.218081 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.218113 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:59:37.218191 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 02:59:37.218332 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2179,"thread_start_us":9164,"threads_started":1}
I20260620 02:59:37.219720 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.202116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41054 (local address 127.0.97.3:44065)
0620 02:59:37.208616 (+ 6500us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:37.208629 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:37.208668 (+ 39us) server_negotiation.cc:407] Connection header received
0620 02:59:37.208794 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:37.208806 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:37.209106 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:37.209329 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:37.210556 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.213216 (+ 2660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.215918 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.217402 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.217530 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.219001 (+ 1471us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:37.219122 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 02:59:37.219307 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6277,"thread_start_us":6114,"threads_started":1}
I20260620 02:59:37.223912 7734 tablet_service.cc:1467] Processing CreateTablet for tablet 78b4782e43c044c6b7a19a556e7aa6fe (DEFAULT_TABLE table=test-table [id=7ec35f78b54540ee9ec70146ebfa0832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:37.225932 7734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b4782e43c044c6b7a19a556e7aa6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:37.226572 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.205604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44039 (local address 127.0.0.1:59298)
0620 02:59:37.217587 (+ 11983us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:37.217606 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:37.217714 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:37.218595 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:37.218612 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:37.218651 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:37.219163 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:37.219174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.222148 (+ 2974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.222154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:37.224323 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:37.224335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.225625 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.225632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:37.225800 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.225834 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 02:59:37.225955 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 02:59:37.226195 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10049,"mutex_wait_us":121,"thread_start_us":9901,"threads_started":1}
I20260620 02:59:37.226464 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.206130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59298 (local address 127.0.97.2:44039)
0620 02:59:37.215375 (+ 9245us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:37.215384 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:37.217847 (+ 2463us) server_negotiation.cc:407] Connection header received
0620 02:59:37.217951 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:37.217961 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:37.218369 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:37.218580 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:37.220234 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.221999 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.224472 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.225407 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.225555 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.225814 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:37.226017 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 02:59:37.226183 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9083,"thread_start_us":8989,"threads_started":1}
I20260620 02:59:37.227825 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.202314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43727 (local address 127.0.0.1:36936)
0620 02:59:37.211531 (+ 9217us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:37.211550 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:37.211655 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:37.219992 (+ 8337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:37.219997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:37.220022 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:37.220362 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:37.220373 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.222148 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.222154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:37.224973 (+ 2819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:37.224987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.226995 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.227003 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:37.227200 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.227231 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 02:59:37.227317 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 02:59:37.227443 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9099,"mutex_wait_us":59,"thread_start_us":1826,"threads_started":1}
I20260620 02:59:37.228617 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.202869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.0.97.1:43727)
0620 02:59:37.219419 (+ 16550us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:37.219428 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:37.219456 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:59:37.219547 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:37.219555 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:37.219820 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:37.220006 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:37.220548 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.221999 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.225164 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.226542 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.226662 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.226848 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:37.228182 (+ 1334us) server_negotiation.cc:299] Negotiation successful
0620 02:59:37.228333 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16376,"thread_start_us":16262,"threads_started":1}
I20260620 02:59:37.230947 7593 tablet_service.cc:1467] Processing CreateTablet for tablet 78b4782e43c044c6b7a19a556e7aa6fe (DEFAULT_TABLE table=test-table [id=7ec35f78b54540ee9ec70146ebfa0832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:37.232407 7593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b4782e43c044c6b7a19a556e7aa6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:37.232177 7445 tablet_service.cc:1467] Processing CreateTablet for tablet 78b4782e43c044c6b7a19a556e7aa6fe (DEFAULT_TABLE table=test-table [id=7ec35f78b54540ee9ec70146ebfa0832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:37.233536 7445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b4782e43c044c6b7a19a556e7aa6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:37.239454 7833 tablet_bootstrap.cc:492] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145: Bootstrap starting.
I20260620 02:59:37.242847 7833 tablet_bootstrap.cc:654] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:37.244047 7833 log.cc:824] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:37.247339 7833 tablet_bootstrap.cc:492] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145: No bootstrap required, opened a new log
I20260620 02:59:37.247634 7833 ts_tablet_manager.cc:1397] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 02:59:37.249235 7835 tablet_bootstrap.cc:492] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073: Bootstrap starting.
I20260620 02:59:37.252846 7835 tablet_bootstrap.cc:654] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:37.255996 7835 log.cc:824] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:37.257583 7836 tablet_bootstrap.cc:492] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d: Bootstrap starting.
I20260620 02:59:37.258674 7833 raft_consensus.cc:348] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } }
I20260620 02:59:37.259230 7833 raft_consensus.cc:374] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:37.259325 7833 raft_consensus.cc:729] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f27e01b9b2cc4d48a7caddbdce674145, State: Initialized, Role: FOLLOWER
I20260620 02:59:37.260218 7833 consensus_queue.cc:260] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } }
I20260620 02:59:37.260931 7836 tablet_bootstrap.cc:654] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:37.262164 7836 log.cc:824] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:37.270131 7833 ts_tablet_manager.cc:1428] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145: Time spent starting tablet: real 0.022s user 0.014s sys 0.010s
I20260620 02:59:37.270387 7800 heartbeater.cc:503] Master 127.0.97.62:35199 was elected leader, sending a full tablet report...
I20260620 02:59:37.277937 7836 tablet_bootstrap.cc:492] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d: No bootstrap required, opened a new log
I20260620 02:59:37.277937 7835 tablet_bootstrap.cc:492] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073: No bootstrap required, opened a new log
I20260620 02:59:37.278244 7836 ts_tablet_manager.cc:1397] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.004s
I20260620 02:59:37.278255 7835 ts_tablet_manager.cc:1397] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.016s
I20260620 02:59:37.286123 7835 raft_consensus.cc:348] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } }
I20260620 02:59:37.286511 7835 raft_consensus.cc:374] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:37.286615 7835 raft_consensus.cc:729] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe8419e2d0a4217880d7b0ca459c073, State: Initialized, Role: FOLLOWER
I20260620 02:59:37.287173 7835 consensus_queue.cc:260] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } }
I20260620 02:59:37.288240 7659 heartbeater.cc:503] Master 127.0.97.62:35199 was elected leader, sending a full tablet report...
I20260620 02:59:37.288122 7836 raft_consensus.cc:348] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } }
I20260620 02:59:37.288524 7836 raft_consensus.cc:374] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:37.288618 7836 raft_consensus.cc:729] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc18659e29e43eea11d6a12dd573c8d, State: Initialized, Role: FOLLOWER
I20260620 02:59:37.288949 7835 ts_tablet_manager.cc:1428] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260620 02:59:37.289148 7836 consensus_queue.cc:260] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } }
I20260620 02:59:37.290858 7836 ts_tablet_manager.cc:1428] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
W20260620 02:59:37.366274 7802 tablet.cc:2369] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:59:37.366796 7840 raft_consensus.cc:482] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:59:37.367093 7840 raft_consensus.cc:504] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } }
I20260620 02:59:37.368928 7840 leader_election.cc:288] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f27e01b9b2cc4d48a7caddbdce674145 (127.0.97.3:44065), 3dc18659e29e43eea11d6a12dd573c8d (127.0.97.1:43727)
I20260620 02:59:37.378423 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.369300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40563 (local address 127.0.97.1:43727)
0620 02:59:37.369500 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:37.369509 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:37.370707 (+ 1198us) server_negotiation.cc:407] Connection header received
0620 02:59:37.370892 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:37.370895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:37.370955 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:37.371032 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:37.371694 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.372807 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.375996 (+ 3189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.377216 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.377353 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.377452 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:37.378034 (+ 582us) server_negotiation.cc:299] Negotiation successful
0620 02:59:37.378165 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:59:37.378423 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.369551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36431 (local address 127.0.97.3:44065)
0620 02:59:37.369792 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:37.369801 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:37.370750 (+ 949us) server_negotiation.cc:407] Connection header received
0620 02:59:37.371171 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:37.371175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:37.371226 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:37.371305 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:37.371881 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.372915 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.375996 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.377216 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.377353 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.377454 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:37.378034 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 02:59:37.378165 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:59:37.378722 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.369105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44065 (local address 127.0.97.2:36431)
0620 02:59:37.370578 (+ 1473us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:37.370600 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:37.370759 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:37.371357 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:37.371362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:37.371384 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:37.371744 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:37.371752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.373041 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.373048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:37.375822 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:37.375837 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.377536 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.377541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:37.377790 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.377892 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 02:59:37.378005 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 02:59:37.378210 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1322,"mutex_wait_us":306,"spinlock_wait_cycles":8448,"thread_start_us":1072,"threads_started":1}
I20260620 02:59:37.379061 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.369105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43727 (local address 127.0.97.2:40563)
0620 02:59:37.370578 (+ 1473us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:37.370600 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:37.370759 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:37.371103 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:37.371106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:37.371133 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:37.371502 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:37.371519 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.372953 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.372956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:37.375822 (+ 2866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:37.375837 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.377521 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.377529 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:37.377854 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.377905 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 02:59:37.378005 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 02:59:37.378227 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1349,"mutex_wait_us":608,"spinlock_wait_cycles":5504,"thread_start_us":812,"threads_started":1}
I20260620 02:59:37.379760 7754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b4782e43c044c6b7a19a556e7aa6fe" candidate_uuid: "0fe8419e2d0a4217880d7b0ca459c073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f27e01b9b2cc4d48a7caddbdce674145" is_pre_election: true
I20260620 02:59:37.379987 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b4782e43c044c6b7a19a556e7aa6fe" candidate_uuid: "0fe8419e2d0a4217880d7b0ca459c073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc18659e29e43eea11d6a12dd573c8d" is_pre_election: true
I20260620 02:59:37.380369 7754 raft_consensus.cc:2454] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe8419e2d0a4217880d7b0ca459c073 in term 0.
I20260620 02:59:37.380508 7465 raft_consensus.cc:2454] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe8419e2d0a4217880d7b0ca459c073 in term 0.
I20260620 02:59:37.381115 7549 leader_election.cc:302] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe8419e2d0a4217880d7b0ca459c073, f27e01b9b2cc4d48a7caddbdce674145; no voters:
I20260620 02:59:37.381526 7840 raft_consensus.cc:2781] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:59:37.381739 7840 raft_consensus.cc:482] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:59:37.381855 7840 raft_consensus.cc:3037] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:37.384232 7840 raft_consensus.cc:504] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } }
I20260620 02:59:37.384802 7840 leader_election.cc:288] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [CANDIDATE]: Term 1 election: Requested vote from peers f27e01b9b2cc4d48a7caddbdce674145 (127.0.97.3:44065), 3dc18659e29e43eea11d6a12dd573c8d (127.0.97.1:43727)
I20260620 02:59:37.385381 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b4782e43c044c6b7a19a556e7aa6fe" candidate_uuid: "0fe8419e2d0a4217880d7b0ca459c073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc18659e29e43eea11d6a12dd573c8d"
I20260620 02:59:37.385368 7754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b4782e43c044c6b7a19a556e7aa6fe" candidate_uuid: "0fe8419e2d0a4217880d7b0ca459c073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f27e01b9b2cc4d48a7caddbdce674145"
I20260620 02:59:37.385628 7754 raft_consensus.cc:3037] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:37.385628 7465 raft_consensus.cc:3037] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:37.389595 7754 raft_consensus.cc:2454] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe8419e2d0a4217880d7b0ca459c073 in term 1.
I20260620 02:59:37.389595 7465 raft_consensus.cc:2454] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe8419e2d0a4217880d7b0ca459c073 in term 1.
I20260620 02:59:37.390077 7546 leader_election.cc:302] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe8419e2d0a4217880d7b0ca459c073, 3dc18659e29e43eea11d6a12dd573c8d; no voters:
I20260620 02:59:37.390395 7840 raft_consensus.cc:2781] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:37.390843 7840 raft_consensus.cc:686] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [term 1 LEADER]: Becoming Leader. State: Replica: 0fe8419e2d0a4217880d7b0ca459c073, State: Running, Role: LEADER
I20260620 02:59:37.391402 7840 consensus_queue.cc:237] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } }
I20260620 02:59:37.396656 6545 catalog_manager.cc:5495] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe8419e2d0a4217880d7b0ca459c073 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0fe8419e2d0a4217880d7b0ca459c073" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } health_report { overall_health: HEALTHY } } }
W20260620 02:59:37.397867 7513 tablet.cc:2369] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:59:37.418110 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.412342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59302 (local address 127.0.97.2:44039)
0620 02:59:37.412514 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:37.412518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:37.412536 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:59:37.412655 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:37.412659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:37.412746 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:37.412858 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:59:37.413195 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.414409 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.415902 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.416297 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.416390 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.416912 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:59:37.417460 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:59:37.417624 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:37.417733 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 02:59:37.417818 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 02:59:37.431321 7465 raft_consensus.cc:1261] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d [term 1 FOLLOWER]: Refusing update from remote peer 0fe8419e2d0a4217880d7b0ca459c073: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:37.431319 7754 raft_consensus.cc:1261] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145 [term 1 FOLLOWER]: Refusing update from remote peer 0fe8419e2d0a4217880d7b0ca459c073: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:37.432397 7844 consensus_queue.cc:1035] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:37.432823 7840 consensus_queue.cc:1035] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:37.444914 7847 mvcc.cc:205] Tried to move back new op lower bound from 7298762249947586560 to 7298762249799000064. Current Snapshot: MvccSnapshot[applied={T|T < 7298762249947586560}]
I20260620 02:59:37.454279 7849 mvcc.cc:205] Tried to move back new op lower bound from 7298762249947586560 to 7298762249799000064. Current Snapshot: MvccSnapshot[applied={T|T < 7298762249947586560}]
I20260620 02:59:37.464323 6544 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:37.464542 6544 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:37.470964 6544 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2e5f36f1afb417f86b54cb6c0ee43a4: Sending DeleteTablet for 3 replicas of tablet 78b4782e43c044c6b7a19a556e7aa6fe
I20260620 02:59:37.472039 7734 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b4782e43c044c6b7a19a556e7aa6fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:37 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41054
I20260620 02:59:37.472342 7593 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b4782e43c044c6b7a19a556e7aa6fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:37 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59298
I20260620 02:59:37.472780 7445 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b4782e43c044c6b7a19a556e7aa6fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:37 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36936
I20260620 02:59:37.513382 7858 tablet_replica.cc:330] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d: stopping tablet replica
I20260620 02:59:37.515187 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 6513
I20260620 02:59:37.516525 6575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:37.518654 7858 raft_consensus.cc:2229] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:37.519335 7858 raft_consensus.cc:2258] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:37.521735 7856 tablet_replica.cc:330] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145: stopping tablet replica
I20260620 02:59:37.526167 7858 ts_tablet_manager.cc:1905] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:37.534005 7856 raft_consensus.cc:2229] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:37.534756 7856 raft_consensus.cc:2258] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:37.540503 7857 tablet_replica.cc:330] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073: stopping tablet replica
I20260620 02:59:37.541081 7857 raft_consensus.cc:2229] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:59:37.544701 7856 ts_tablet_manager.cc:1905] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:37.545857 7857 raft_consensus.cc:2258] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:37.551654 7858 ts_tablet_manager.cc:1918] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:59:37.551848 7858 log.cc:1196] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/78b4782e43c044c6b7a19a556e7aa6fe
I20260620 02:59:37.552485 7858 ts_tablet_manager.cc:1939] T 78b4782e43c044c6b7a19a556e7aa6fe P 3dc18659e29e43eea11d6a12dd573c8d: Deleting consensus metadata
I20260620 02:59:37.556138 7857 ts_tablet_manager.cc:1905] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:37.561846 7856 ts_tablet_manager.cc:1918] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:59:37.562031 7856 log.cc:1196] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/78b4782e43c044c6b7a19a556e7aa6fe
I20260620 02:59:37.562534 7856 ts_tablet_manager.cc:1939] T 78b4782e43c044c6b7a19a556e7aa6fe P f27e01b9b2cc4d48a7caddbdce674145: Deleting consensus metadata
I20260620 02:59:37.566599 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.474350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54236 (local address 127.0.97.61:43161)
0620 02:59:37.474514 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:37.474520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:37.514399 (+ 39879us) server_negotiation.cc:407] Connection header received
0620 02:59:37.515200 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:37.515209 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:37.515303 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:37.515387 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:37.523330 (+ 7943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.524861 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.542555 (+ 17694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.543320 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.543422 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.551831 (+ 8409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:37.551840 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:37.551848 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:37.551905 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:37.556368 (+ 4463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.557422 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.557429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.557433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.557615 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.564340 (+ 6725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.564347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.564350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.564763 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:37.565001 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:37.565722 (+ 721us) server_negotiation.cc:299] Negotiation successful
0620 02:59:37.566067 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 02:59:37.570366 7857 ts_tablet_manager.cc:1918] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:59:37.575227 7857 log.cc:1196] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/78b4782e43c044c6b7a19a556e7aa6fe
I20260620 02:59:37.575871 7857 ts_tablet_manager.cc:1939] T 78b4782e43c044c6b7a19a556e7aa6fe P 0fe8419e2d0a4217880d7b0ca459c073: Deleting consensus metadata
I20260620 02:59:37.585024 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.514674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51352 (local address 127.0.97.60:34735)
0620 02:59:37.514873 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:37.514879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:37.569078 (+ 54199us) server_negotiation.cc:407] Connection header received
0620 02:59:37.569329 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:37.569336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:37.569420 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:37.569489 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:37.571465 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.572665 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.574733 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.575463 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.575608 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.579078 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:37.579084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:37.579087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:37.579119 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:37.582095 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.582906 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.582911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.582915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.583093 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.583518 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.583528 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.583530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.583826 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:37.584100 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:37.584430 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 02:59:37.584662 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 02:59:37.723696 6532 catalog_manager.cc:4841] TS f27e01b9b2cc4d48a7caddbdce674145 (127.0.97.3:44065): tablet 78b4782e43c044c6b7a19a556e7aa6fe (table test-table [id=7ec35f78b54540ee9ec70146ebfa0832]) successfully deleted
I20260620 02:59:37.723676 6529 catalog_manager.cc:4841] TS 0fe8419e2d0a4217880d7b0ca459c073 (127.0.97.2:44039): tablet 78b4782e43c044c6b7a19a556e7aa6fe (table test-table [id=7ec35f78b54540ee9ec70146ebfa0832]) successfully deleted
I20260620 02:59:37.724152 6529 catalog_manager.cc:4841] TS 3dc18659e29e43eea11d6a12dd573c8d (127.0.97.1:43727): tablet 78b4782e43c044c6b7a19a556e7aa6fe (table test-table [id=7ec35f78b54540ee9ec70146ebfa0832]) successfully deleted
I20260620 02:59:37.726498 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 6513
I20260620 02:59:37.788376 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.761451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51366 (local address 127.0.97.60:34735)
0620 02:59:37.761932 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:37.761941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:37.769390 (+ 7449us) server_negotiation.cc:407] Connection header received
0620 02:59:37.769894 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:37.769903 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:37.770005 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:37.770103 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:37.773443 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.774896 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.777768 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.778667 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.778789 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.782764 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:37.782769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:37.782772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:37.782805 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:37.785442 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.786285 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.786291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.786295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.786438 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.786815 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.786820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.786824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.787149 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:37.787374 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:37.787914 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 02:59:37.788125 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316}
I20260620 02:59:37.788475 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:37.761690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54248 (local address 127.0.97.61:43161)
0620 02:59:37.762119 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:37.762127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:37.768800 (+ 6673us) server_negotiation.cc:407] Connection header received
0620 02:59:37.769315 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:37.769321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:37.769402 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:37.769474 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:37.771719 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.772823 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.774850 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:37.775489 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:37.775589 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:37.781134 (+ 5545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:37.781143 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:37.781150 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:37.781203 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:37.785115 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.786079 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.786085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.786088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.786236 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:37.786665 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:37.786671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:37.786675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:37.787051 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:37.787305 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:37.787688 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 02:59:37.788094 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260620 02:59:38.454200 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:38.453485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.1:59369)
0620 02:59:38.453947 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:38.454053 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":246,"threads_started":1}
W20260620 02:59:38.454646 7511 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35199 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
I20260620 02:59:38.457988 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:38.456873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.2:43121)
0620 02:59:38.457668 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:38.457795 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":547,"threads_started":1}
W20260620 02:59:38.458367 7659 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35199 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
I20260620 02:59:38.459796 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:38.458928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.3:56855)
0620 02:59:38.459423 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:38.459585 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":230,"threads_started":1}
W20260620 02:59:38.460114 7800 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35199 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
I20260620 02:59:38.971726 7871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c2e5f36f1afb417f86b54cb6c0ee43a4)
I20260620 02:59:38.971843 7872 raft_consensus.cc:482] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c2e5f36f1afb417f86b54cb6c0ee43a4)
I20260620 02:59:38.971973 7871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: false } }
I20260620 02:59:38.972179 7872 raft_consensus.cc:504] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: false } }
I20260620 02:59:38.972904 7871 leader_election.cc:288] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2e5f36f1afb417f86b54cb6c0ee43a4 (127.0.97.62:35199), d140ec2db4674ef1ade56ccc44494c5d (127.0.97.60:34735)
I20260620 02:59:38.988881 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:38.973162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.0.1:60614)
0620 02:59:38.988274 (+ 15112us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:38.988494 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14966,"thread_start_us":8551,"threads_started":1}
W20260620 02:59:38.989180 7128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
I20260620 02:59:38.989904 7872 leader_election.cc:288] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2e5f36f1afb417f86b54cb6c0ee43a4 (127.0.97.62:35199), 683840b125cc4cf791dc563ae6126dc9 (127.0.97.61:43161)
I20260620 02:59:38.991664 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:38.973130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34735 (local address 127.0.0.1:51380)
0620 02:59:38.981297 (+ 8167us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:38.981324 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:38.981505 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:38.982957 (+ 1452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:38.982964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:38.983011 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:38.983401 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:38.983418 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:38.985689 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:38.985702 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:38.988618 (+ 2916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:38.988638 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:38.990791 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:38.990797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:38.990991 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:38.991037 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 02:59:38.991125 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 02:59:38.991278 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8004,"thread_start_us":16390,"threads_started":1}
I20260620 02:59:38.992890 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:38.973540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51380 (local address 127.0.97.60:34735)
0620 02:59:38.982556 (+ 9016us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:38.982562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:38.982579 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:59:38.982652 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:38.982657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:38.982762 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:38.983597 (+ 835us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:38.983845 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:38.985427 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:38.988863 (+ 3436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:38.990606 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:38.992238 (+ 1632us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:38.992339 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:38.992386 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 02:59:38.992507 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8898,"thread_start_us":8698,"threads_started":1}
I20260620 02:59:38.993700 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:38.982586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.0.1:60622)
0620 02:59:38.993166 (+ 10580us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:38.993375 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10445,"thread_start_us":6733,"threads_started":1}
I20260620 02:59:38.993758 7340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "683840b125cc4cf791dc563ae6126dc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d140ec2db4674ef1ade56ccc44494c5d" is_pre_election: true
W20260620 02:59:38.994058 7315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
I20260620 02:59:38.994252 7340 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 683840b125cc4cf791dc563ae6126dc9 in term 1.
I20260620 02:59:38.994671 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:38.994173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.0.1:60638)
0620 02:59:38.994442 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:38.994530 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
W20260620 02:59:38.994853 7128 leader_election.cc:334] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c2e5f36f1afb417f86b54cb6c0ee43a4 (127.0.97.62:35199): Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
I20260620 02:59:38.995126 7127 leader_election.cc:302] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 683840b125cc4cf791dc563ae6126dc9, d140ec2db4674ef1ade56ccc44494c5d; no voters: c2e5f36f1afb417f86b54cb6c0ee43a4
I20260620 02:59:38.995464 7871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 02:59:38.995571 7871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c2e5f36f1afb417f86b54cb6c0ee43a4)
I20260620 02:59:38.995694 7871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:59:38.995860 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:38.995445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.0.1:60646)
0620 02:59:38.995642 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:38.995731 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260620 02:59:38.996049 7315 leader_election.cc:334] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c2e5f36f1afb417f86b54cb6c0ee43a4 (127.0.97.62:35199): Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
I20260620 02:59:38.998284 7871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: false } }
I20260620 02:59:38.998940 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:38.974134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43161 (local address 127.0.0.1:54256)
0620 02:59:38.989085 (+ 14951us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:38.989104 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:38.989214 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:38.990783 (+ 1569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:38.990788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:38.990815 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:38.991124 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:38.991137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:38.993674 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:38.993681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:38.996763 (+ 3082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:38.996774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:38.998158 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:38.998164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:38.998350 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:38.998373 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 02:59:38.998477 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 02:59:38.998603 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14822,"thread_start_us":18766,"threads_started":1}
I20260620 02:59:38.998999 7871 leader_election.cc:288] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [CANDIDATE]: Term 2 election: Requested vote from peers c2e5f36f1afb417f86b54cb6c0ee43a4 (127.0.97.62:35199), d140ec2db4674ef1ade56ccc44494c5d (127.0.97.60:34735)
I20260620 02:59:38.999030 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:38.974573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54256 (local address 127.0.97.61:43161)
0620 02:59:38.990475 (+ 15902us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:38.990481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:38.990498 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:59:38.990552 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:38.990559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:38.990640 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:38.991267 (+ 627us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:38.991492 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:38.992717 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:38.996918 (+ 4201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:38.997985 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:38.998127 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:38.998249 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:38.998500 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 02:59:38.998626 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15785,"thread_start_us":15604,"threads_started":1}
I20260620 02:59:38.999544 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:38.999010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.0.1:60660)
0620 02:59:38.999333 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:38.999411 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217}
I20260620 02:59:38.999580 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d140ec2db4674ef1ade56ccc44494c5d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "683840b125cc4cf791dc563ae6126dc9" is_pre_election: true
I20260620 02:59:38.999756 7340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "683840b125cc4cf791dc563ae6126dc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d140ec2db4674ef1ade56ccc44494c5d"
I20260620 02:59:38.999946 7153 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d140ec2db4674ef1ade56ccc44494c5d in current term 2: Already voted for candidate 683840b125cc4cf791dc563ae6126dc9 in this term.
I20260620 02:59:38.999977 7340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:59:39.000480 7317 leader_election.cc:302] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d140ec2db4674ef1ade56ccc44494c5d; no voters: 683840b125cc4cf791dc563ae6126dc9, c2e5f36f1afb417f86b54cb6c0ee43a4
I20260620 02:59:39.000560 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.000168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.0.1:60668)
0620 02:59:39.000352 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.000428 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260620 02:59:39.000742 7128 leader_election.cc:334] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c2e5f36f1afb417f86b54cb6c0ee43a4 (127.0.97.62:35199): Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
I20260620 02:59:39.002557 7340 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 683840b125cc4cf791dc563ae6126dc9 in term 2.
I20260620 02:59:39.002784 7872 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:59:39.002934 7127 leader_election.cc:302] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 683840b125cc4cf791dc563ae6126dc9, d140ec2db4674ef1ade56ccc44494c5d; no voters: c2e5f36f1afb417f86b54cb6c0ee43a4
I20260620 02:59:39.003232 7871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:59:39.003605 7871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [term 2 LEADER]: Becoming Leader. State: Replica: 683840b125cc4cf791dc563ae6126dc9, State: Running, Role: LEADER
I20260620 02:59:39.003965 7871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: false } }
I20260620 02:59:39.006158 7881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 683840b125cc4cf791dc563ae6126dc9. Latest consensus state: current_term: 2 leader_uuid: "683840b125cc4cf791dc563ae6126dc9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: false } } }
I20260620 02:59:39.006435 7881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:39.007215 7883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:59:39.011180 7883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:59:39.012125 7883 catalog_manager.cc:1259] Loaded cluster ID: 437db21f6b864dab8a24edfc3a3cdbeb
I20260620 02:59:39.012192 7883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:59:39.013327 7883 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:59:39.014192 7883 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: Loaded TSK: 0
I20260620 02:59:39.014998 7883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:59:39.066130 7143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:59:39.072979 7143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:59:39.077242 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.076815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.0.1:60700)
0620 02:59:39.076982 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.077071 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 02:59:39.077327 7340 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [term 2 FOLLOWER]: Refusing update from remote peer 683840b125cc4cf791dc563ae6126dc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 02:59:39.078056 7881 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 02:59:39.078301 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.077927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.0.1:60712)
0620 02:59:39.078077 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.078176 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 02:59:39.078549 7128 consensus_peers.cc:487] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 -> Peer c2e5f36f1afb417f86b54cb6c0ee43a4 (127.0.97.62:35199): Couldn't send request to peer c2e5f36f1afb417f86b54cb6c0ee43a4. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 02:59:39.080633 7872 sys_catalog.cc:455] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 683840b125cc4cf791dc563ae6126dc9. Latest consensus state: current_term: 2 leader_uuid: "683840b125cc4cf791dc563ae6126dc9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: false } } }
I20260620 02:59:39.080899 7872 sys_catalog.cc:458] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:59:39.082549 7881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "683840b125cc4cf791dc563ae6126dc9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: false } } }
I20260620 02:59:39.082862 7881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:39.082875 7872 sys_catalog.cc:455] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "683840b125cc4cf791dc563ae6126dc9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c2e5f36f1afb417f86b54cb6c0ee43a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } peers { permanent_uuid: "683840b125cc4cf791dc563ae6126dc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43161 } attrs { promote: false } } peers { permanent_uuid: "d140ec2db4674ef1ade56ccc44494c5d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34735 } attrs { promote: false } } }
I20260620 02:59:39.083105 7872 sys_catalog.cc:458] T 00000000000000000000000000000000 P d140ec2db4674ef1ade56ccc44494c5d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:59:39.100292 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.092219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36940 (local address 127.0.97.1:43727)
0620 02:59:39.092624 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:39.092630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:39.092647 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:59:39.092711 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:39.092715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:39.092785 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:39.092926 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:39.093520 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.095362 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.098391 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.099305 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.099458 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:39.099564 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:39.099849 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 02:59:39.099981 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":229,"threads_started":1}
I20260620 02:59:39.100283 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.091748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43727 (local address 127.0.0.1:36940)
0620 02:59:39.091900 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.091918 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:39.092019 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:39.092970 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:39.092978 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:39.093007 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:39.093357 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:39.093370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.095502 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.095508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:39.098184 (+ 2676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:39.098205 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.099506 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.099513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:39.099686 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:39.099709 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 02:59:39.099825 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 02:59:39.099969 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 02:59:39.101326 7445 tablet_service.cc:1467] Processing CreateTablet for tablet 73903d27bf7443d7821c44bd35b9ad0f (DEFAULT_TABLE table=test-table [id=0a5ad78e5ad04087b0b347bb67af1863]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:39.101536 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.090888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44065 (local address 127.0.0.1:41062)
0620 02:59:39.091104 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.091123 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:39.091245 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:39.091705 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:39.091711 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:39.091737 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:39.091990 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:39.092002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.095620 (+ 3618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.095629 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:39.098347 (+ 2718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:39.098367 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.100519 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.100528 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:39.100803 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:39.100859 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 02:59:39.100934 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 02:59:39.101135 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":1}
I20260620 02:59:39.101842 7445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73903d27bf7443d7821c44bd35b9ad0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:39.102761 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.090951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41062 (local address 127.0.97.3:44065)
0620 02:59:39.091410 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:39.091414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:39.091429 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:59:39.091507 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:39.091510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:39.091564 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:39.091676 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:39.092586 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.095408 (+ 2822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.098609 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.100288 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.102195 (+ 1907us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:39.102308 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:39.102361 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 02:59:39.102497 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":275,"threads_started":1}
I20260620 02:59:39.103406 7734 tablet_service.cc:1467] Processing CreateTablet for tablet 73903d27bf7443d7821c44bd35b9ad0f (DEFAULT_TABLE table=test-table [id=0a5ad78e5ad04087b0b347bb67af1863]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:39.103809 7734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73903d27bf7443d7821c44bd35b9ad0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:39.108664 7894 tablet_bootstrap.cc:492] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d: Bootstrap starting.
I20260620 02:59:39.110252 7895 tablet_bootstrap.cc:492] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145: Bootstrap starting.
I20260620 02:59:39.110582 7894 tablet_bootstrap.cc:654] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:39.111267 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.092062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44039 (local address 127.0.0.1:59316)
0620 02:59:39.092596 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.092614 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:39.092724 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:39.105230 (+ 12506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:39.105234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:39.105260 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:39.105615 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:39.105630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.106640 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.106649 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:39.108597 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:39.108611 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.110424 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.110433 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:39.110628 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:39.110663 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:59:39.110757 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 02:59:39.110918 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":458,"threads_started":1}
I20260620 02:59:39.111678 7895 tablet_bootstrap.cc:654] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:39.112039 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.092409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59316 (local address 127.0.97.2:44039)
0620 02:59:39.104948 (+ 12539us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:39.104952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:39.104968 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:59:39.105026 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:39.105029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:39.105090 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:39.105212 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:39.105817 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.106504 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.108789 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.110034 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.110158 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:39.110265 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:39.111625 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0620 02:59:39.111780 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12438,"thread_start_us":12962,"threads_started":1}
I20260620 02:59:39.112890 7593 tablet_service.cc:1467] Processing CreateTablet for tablet 73903d27bf7443d7821c44bd35b9ad0f (DEFAULT_TABLE table=test-table [id=0a5ad78e5ad04087b0b347bb67af1863]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:39.113294 7894 tablet_bootstrap.cc:492] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d: No bootstrap required, opened a new log
I20260620 02:59:39.113334 7593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73903d27bf7443d7821c44bd35b9ad0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:39.113528 7894 ts_tablet_manager.cc:1397] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 02:59:39.114156 7895 tablet_bootstrap.cc:492] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145: No bootstrap required, opened a new log
I20260620 02:59:39.114289 7895 ts_tablet_manager.cc:1397] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 02:59:39.114490 7894 raft_consensus.cc:348] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } }
I20260620 02:59:39.114710 7894 raft_consensus.cc:374] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:39.114811 7894 raft_consensus.cc:729] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc18659e29e43eea11d6a12dd573c8d, State: Initialized, Role: FOLLOWER
I20260620 02:59:39.115010 7895 raft_consensus.cc:348] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } }
I20260620 02:59:39.115085 7894 consensus_queue.cc:260] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } }
I20260620 02:59:39.115247 7895 raft_consensus.cc:374] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:39.115341 7895 raft_consensus.cc:729] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f27e01b9b2cc4d48a7caddbdce674145, State: Initialized, Role: FOLLOWER
I20260620 02:59:39.115602 7895 consensus_queue.cc:260] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } }
I20260620 02:59:39.116109 7894 ts_tablet_manager.cc:1428] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 02:59:39.118829 7512 heartbeater.cc:503] Master 127.0.97.61:43161 was elected leader, sending a full tablet report...
I20260620 02:59:39.122149 7900 tablet_bootstrap.cc:492] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073: Bootstrap starting.
I20260620 02:59:39.122699 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.116161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.1:36793)
0620 02:59:39.122050 (+ 5889us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.122361 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5743,"thread_start_us":5611,"threads_started":1}
I20260620 02:59:39.123772 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.123282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.1:46551)
0620 02:59:39.123452 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.123557 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 02:59:39.123910 7900 tablet_bootstrap.cc:654] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:39.126121 7900 tablet_bootstrap.cc:492] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073: No bootstrap required, opened a new log
I20260620 02:59:39.126165 7895 ts_tablet_manager.cc:1428] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145: Time spent starting tablet: real 0.012s user 0.000s sys 0.005s
I20260620 02:59:39.126267 7900 ts_tablet_manager.cc:1397] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260620 02:59:39.128474 7801 heartbeater.cc:503] Master 127.0.97.61:43161 was elected leader, sending a full tablet report...
I20260620 02:59:39.128500 7900 raft_consensus.cc:348] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } }
I20260620 02:59:39.128727 7900 raft_consensus.cc:374] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:39.128871 7900 raft_consensus.cc:729] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe8419e2d0a4217880d7b0ca459c073, State: Initialized, Role: FOLLOWER
I20260620 02:59:39.130200 7900 consensus_queue.cc:260] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } }
I20260620 02:59:39.131354 7900 ts_tablet_manager.cc:1428] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 02:59:39.132047 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.126577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.3:33565)
0620 02:59:39.131669 (+ 5092us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.131800 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4234,"thread_start_us":4096,"threads_started":1}
I20260620 02:59:39.132090 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.131260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.2:42469)
0620 02:59:39.131784 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.131892 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"threads_started":1}
I20260620 02:59:39.132812 7660 heartbeater.cc:503] Master 127.0.97.61:43161 was elected leader, sending a full tablet report...
I20260620 02:59:39.133416 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.133019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.2:42143)
0620 02:59:39.133206 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.133283 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 02:59:39.133620 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.133179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.3:45627)
0620 02:59:39.133377 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.133463 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260620 02:59:39.282408 7661 tablet.cc:2369] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:59:39.339412 7904 raft_consensus.cc:482] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:59:39.339922 7904 raft_consensus.cc:504] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } }
I20260620 02:59:39.340744 7904 leader_election.cc:288] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f27e01b9b2cc4d48a7caddbdce674145 (127.0.97.3:44065), 3dc18659e29e43eea11d6a12dd573c8d (127.0.97.1:43727)
I20260620 02:59:39.341305 7754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73903d27bf7443d7821c44bd35b9ad0f" candidate_uuid: "0fe8419e2d0a4217880d7b0ca459c073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f27e01b9b2cc4d48a7caddbdce674145" is_pre_election: true
I20260620 02:59:39.341373 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73903d27bf7443d7821c44bd35b9ad0f" candidate_uuid: "0fe8419e2d0a4217880d7b0ca459c073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc18659e29e43eea11d6a12dd573c8d" is_pre_election: true
I20260620 02:59:39.341539 7754 raft_consensus.cc:2454] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe8419e2d0a4217880d7b0ca459c073 in term 0.
I20260620 02:59:39.341728 7465 raft_consensus.cc:2454] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe8419e2d0a4217880d7b0ca459c073 in term 0.
I20260620 02:59:39.342065 7549 leader_election.cc:302] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe8419e2d0a4217880d7b0ca459c073, f27e01b9b2cc4d48a7caddbdce674145; no voters:
I20260620 02:59:39.342337 7904 raft_consensus.cc:2781] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:59:39.342469 7904 raft_consensus.cc:482] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:59:39.342583 7904 raft_consensus.cc:3037] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:39.344653 7904 raft_consensus.cc:504] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } }
I20260620 02:59:39.345268 7904 leader_election.cc:288] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [CANDIDATE]: Term 1 election: Requested vote from peers f27e01b9b2cc4d48a7caddbdce674145 (127.0.97.3:44065), 3dc18659e29e43eea11d6a12dd573c8d (127.0.97.1:43727)
I20260620 02:59:39.345860 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73903d27bf7443d7821c44bd35b9ad0f" candidate_uuid: "0fe8419e2d0a4217880d7b0ca459c073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc18659e29e43eea11d6a12dd573c8d"
I20260620 02:59:39.345857 7754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73903d27bf7443d7821c44bd35b9ad0f" candidate_uuid: "0fe8419e2d0a4217880d7b0ca459c073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f27e01b9b2cc4d48a7caddbdce674145"
I20260620 02:59:39.346062 7465 raft_consensus.cc:3037] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:39.346093 7754 raft_consensus.cc:3037] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:39.348026 7465 raft_consensus.cc:2454] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe8419e2d0a4217880d7b0ca459c073 in term 1.
I20260620 02:59:39.348044 7754 raft_consensus.cc:2454] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe8419e2d0a4217880d7b0ca459c073 in term 1.
I20260620 02:59:39.348506 7546 leader_election.cc:302] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0fe8419e2d0a4217880d7b0ca459c073, 3dc18659e29e43eea11d6a12dd573c8d, f27e01b9b2cc4d48a7caddbdce674145; no voters:
I20260620 02:59:39.348809 7904 raft_consensus.cc:2781] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:39.349236 7904 raft_consensus.cc:686] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [term 1 LEADER]: Becoming Leader. State: Replica: 0fe8419e2d0a4217880d7b0ca459c073, State: Running, Role: LEADER
I20260620 02:59:39.349517 7904 consensus_queue.cc:237] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } }
I20260620 02:59:39.350289 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.349841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.2:59785)
0620 02:59:39.349984 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.350098 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 02:59:39.352541 7142 catalog_manager.cc:5495] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe8419e2d0a4217880d7b0ca459c073 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0fe8419e2d0a4217880d7b0ca459c073" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fe8419e2d0a4217880d7b0ca459c073" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44039 } health_report { overall_health: HEALTHY } } }
I20260620 02:59:39.394407 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.390033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.0.97.2:44039)
0620 02:59:39.390222 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:39.390227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:39.390244 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:59:39.390358 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:39.390362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:39.390426 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:39.390499 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:59:39.390864 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.391574 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.392827 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.393236 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.393335 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:39.393738 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:59:39.393845 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:59:39.393911 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:39.394035 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 02:59:39.394094 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:59:39.397400 7465 raft_consensus.cc:1261] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d [term 1 FOLLOWER]: Refusing update from remote peer 0fe8419e2d0a4217880d7b0ca459c073: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:39.397400 7754 raft_consensus.cc:1261] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145 [term 1 FOLLOWER]: Refusing update from remote peer 0fe8419e2d0a4217880d7b0ca459c073: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:39.398113 7906 consensus_queue.cc:1035] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27e01b9b2cc4d48a7caddbdce674145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:39.398443 7904 consensus_queue.cc:1035] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc18659e29e43eea11d6a12dd573c8d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:39.400995 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.400517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.1:48071)
0620 02:59:39.400667 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.400771 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 02:59:39.401070 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.400508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.3:53369)
0620 02:59:39.400693 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.400840 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 02:59:39.406904 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.406432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.3:59245)
0620 02:59:39.406627 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.406723 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 02:59:39.408339 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.407803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.2:46635)
0620 02:59:39.408045 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.408168 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260620 02:59:39.410032 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.409428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.1:52391)
0620 02:59:39.409727 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.409831 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260620 02:59:39.410624 7910 mvcc.cc:205] Tried to move back new op lower bound from 7298762258005614592 to 7298762257819324416. Current Snapshot: MvccSnapshot[applied={T|T < 7298762258005614592}]
I20260620 02:59:39.414784 7142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:39.415078 7142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:39.421063 7142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 683840b125cc4cf791dc563ae6126dc9: Sending DeleteTablet for 3 replicas of tablet 73903d27bf7443d7821c44bd35b9ad0f
I20260620 02:59:39.421964 7734 tablet_service.cc:1514] Processing DeleteTablet for tablet 73903d27bf7443d7821c44bd35b9ad0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:39 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:41062
I20260620 02:59:39.422188 7733 tablet_service.cc:1514] Processing DeleteTablet for tablet 73903d27bf7443d7821c44bd35b9ad0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:39 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:41062
I20260620 02:59:39.422889 7917 tablet_replica.cc:330] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145: stopping tablet replica
I20260620 02:59:39.423081 7917 raft_consensus.cc:2229] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:39.423367 7917 raft_consensus.cc:2258] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:39.423717 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 7351
I20260620 02:59:39.424829 7917 ts_tablet_manager.cc:1905] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:39.425100 7445 tablet_service.cc:1514] Processing DeleteTablet for tablet 73903d27bf7443d7821c44bd35b9ad0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:39 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:36940
I20260620 02:59:39.426436 7593 tablet_service.cc:1514] Processing DeleteTablet for tablet 73903d27bf7443d7821c44bd35b9ad0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:39 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:59316
I20260620 02:59:39.432740 7917 ts_tablet_manager.cc:1918] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:59:39.432904 7917 log.cc:1196] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/73903d27bf7443d7821c44bd35b9ad0f
I20260620 02:59:39.433389 7917 ts_tablet_manager.cc:1939] T 73903d27bf7443d7821c44bd35b9ad0f P f27e01b9b2cc4d48a7caddbdce674145: Deleting consensus metadata
I20260620 02:59:39.434736 7130 catalog_manager.cc:4841] TS f27e01b9b2cc4d48a7caddbdce674145 (127.0.97.3:44065): tablet 73903d27bf7443d7821c44bd35b9ad0f (table test-table [id=0a5ad78e5ad04087b0b347bb67af1863]) successfully deleted
W20260620 02:59:39.435357 7130 catalog_manager.cc:4808] TS f27e01b9b2cc4d48a7caddbdce674145 (127.0.97.3:44065): delete failed for tablet 73903d27bf7443d7821c44bd35b9ad0f because the tablet was not found. No further retry: Not found: Tablet not found: 73903d27bf7443d7821c44bd35b9ad0f
I20260620 02:59:39.436538 7919 tablet_replica.cc:330] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d: stopping tablet replica
I20260620 02:59:39.436717 7919 raft_consensus.cc:2229] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:39.436973 7919 raft_consensus.cc:2258] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:39.439035 7919 ts_tablet_manager.cc:1905] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:39.441975 7920 tablet_replica.cc:330] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073: stopping tablet replica
I20260620 02:59:39.442165 7920 raft_consensus.cc:2229] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:59:39.442560 7920 raft_consensus.cc:2258] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:39.443567 7920 ts_tablet_manager.cc:1905] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:39.447355 7919 ts_tablet_manager.cc:1918] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:59:39.447485 7919 log.cc:1196] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/73903d27bf7443d7821c44bd35b9ad0f
I20260620 02:59:39.447896 7919 ts_tablet_manager.cc:1939] T 73903d27bf7443d7821c44bd35b9ad0f P 3dc18659e29e43eea11d6a12dd573c8d: Deleting consensus metadata
I20260620 02:59:39.448946 7127 catalog_manager.cc:4841] TS 3dc18659e29e43eea11d6a12dd573c8d (127.0.97.1:43727): tablet 73903d27bf7443d7821c44bd35b9ad0f (table test-table [id=0a5ad78e5ad04087b0b347bb67af1863]) successfully deleted
I20260620 02:59:39.450300 7920 ts_tablet_manager.cc:1918] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:59:39.450461 7920 log.cc:1196] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/73903d27bf7443d7821c44bd35b9ad0f
I20260620 02:59:39.450834 7920 ts_tablet_manager.cc:1939] T 73903d27bf7443d7821c44bd35b9ad0f P 0fe8419e2d0a4217880d7b0ca459c073: Deleting consensus metadata
I20260620 02:59:39.451762 7127 catalog_manager.cc:4841] TS 0fe8419e2d0a4217880d7b0ca459c073 (127.0.97.2:44039): tablet 73903d27bf7443d7821c44bd35b9ad0f (table test-table [id=0a5ad78e5ad04087b0b347bb67af1863]) successfully deleted
I20260620 02:59:39.457907 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.424942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36944 (local address 127.0.97.1:43727)
0620 02:59:39.425168 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:39.425174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:39.443276 (+ 18102us) server_negotiation.cc:407] Connection header received
0620 02:59:39.443468 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:39.443473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:39.443558 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:39.443672 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:39.445685 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.446809 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.448846 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.449487 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.449609 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:39.452484 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:39.452498 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:39.452515 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:39.452550 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:39.455415 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:39.456014 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:39.456021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:39.456023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:39.456136 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:39.456513 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:39.456520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:39.456527 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:39.456865 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:39.457083 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:39.457342 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 02:59:39.457611 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 02:59:39.458570 7505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:39.525267 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.524770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.0.1:60724)
0620 02:59:39.524984 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.525087 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 02:59:39.526283 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.525938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.0.1:60738)
0620 02:59:39.526077 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:39.526161 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 02:59:39.561858 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 7351
I20260620 02:59:39.578876 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 7522
I20260620 02:59:39.591822 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.579603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59336 (local address 127.0.97.2:44039)
0620 02:59:39.579937 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:39.579941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:39.579960 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:59:39.580024 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:39.580027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:39.580091 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:39.580176 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:39.581395 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.582154 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.583538 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.583915 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.584006 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:39.586576 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:39.586590 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:39.586607 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:39.586647 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:39.589453 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:39.590101 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:39.590108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:39.590110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:39.590227 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:39.590511 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:39.590514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:39.590516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:39.590859 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:39.591066 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:39.591308 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 02:59:39.591455 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260620 02:59:39.592490 7653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:39.711279 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 7522
I20260620 02:59:39.733368 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 7671
I20260620 02:59:39.750769 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:39.734164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.0.97.3:44065)
0620 02:59:39.734607 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:39.734611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:39.734627 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:59:39.734680 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:39.734684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:39.734741 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:39.734845 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:39.736125 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.736840 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.738076 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:39.738469 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:39.738564 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:39.745158 (+ 6594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:39.745172 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:39.745189 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:39.745229 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:39.747971 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:39.748629 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:39.748637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:39.748640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:39.748766 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:39.749058 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:39.749061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:39.749063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:39.749397 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:39.749682 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:39.750235 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 02:59:39.750412 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":234,"threads_started":1}
I20260620 02:59:39.751394 7794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:39.862071 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 7671
I20260620 02:59:39.879328 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 7006
I20260620 02:59:39.880383 7173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:40.001902 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 7006
W20260620 02:59:40.023113 7315 connection.cc:535] server connection from 127.0.0.1:51380 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 02:59:40.023505 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 7105
I20260620 02:59:40.024525 7361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:40.142668 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 7105
2026-06-20T02:59:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9588 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:59:40 dist-test-slave-f1j0 krb5kdc[7929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:59:40 dist-test-slave-f1j0 krb5kdc[7929](info): set up 2 sockets
Jun 20 02:59:40 dist-test-slave-f1j0 krb5kdc[7929](info): commencing operation
krb5kdc: starting...
W20260620 02:59:41.306478 388 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 02:59:41 dist-test-slave-f1j0 krb5kdc[7929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:59:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:59:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 02:59:41.519449 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45219
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:41.836663 7942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:41.836927 7942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:41.837014 7942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:41.844066 7942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:59:41.844162 7942 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 02:59:41.844208 7942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:41.844273 7942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:41.844321 7942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:59:41.844380 7942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:59:41.853930 7942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45219
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:41.855121 7942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:41.856380 7942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:41.868036 7948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:41.867992 7947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:41.868471 7950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:41.869338 7942 server_base.cc:1034] running on GCE node
I20260620 02:59:41.870085 7942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:41.871393 7942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:41.872619 7942 hybrid_clock.cc:648] HybridClock initialized: now 1781924381872558 us; error 31 us; skew 500 ppm
I20260620 02:59:41.872962 7942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:41 dist-test-slave-f1j0 krb5kdc[7929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924381, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:41.876051 7942 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 02:59:41.877470 7942 webserver.cc:460] Webserver started at http://127.0.97.62:36647/ using document root <none> and password file <none>
I20260620 02:59:41.878176 7942 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:41.878274 7942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:41.878545 7942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:41.881337 7942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "e8bb4209e67c43ddb0af4b1500f7d16d"
format_stamp: "Formatted at 2026-06-20 02:59:41 on dist-test-slave-f1j0"
server_key: "ba18060708f263dcd1e458b2bd0f184a"
server_key_iv: "3975eda90b574161201d378deb6e2cc3"
server_key_version: "encryptionkey@0"
I20260620 02:59:41.882169 7942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "e8bb4209e67c43ddb0af4b1500f7d16d"
format_stamp: "Formatted at 2026-06-20 02:59:41 on dist-test-slave-f1j0"
server_key: "ba18060708f263dcd1e458b2bd0f184a"
server_key_iv: "3975eda90b574161201d378deb6e2cc3"
server_key_version: "encryptionkey@0"
I20260620 02:59:41.887082 7942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:59:41.890453 7957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:41.891757 7942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:59:41.891942 7942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "e8bb4209e67c43ddb0af4b1500f7d16d"
format_stamp: "Formatted at 2026-06-20 02:59:41 on dist-test-slave-f1j0"
server_key: "ba18060708f263dcd1e458b2bd0f184a"
server_key_iv: "3975eda90b574161201d378deb6e2cc3"
server_key_version: "encryptionkey@0"
I20260620 02:59:41.892159 7942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:41.921739 7942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:41.925663 7942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:41.925923 7942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:41.946847 7942 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45219
I20260620 02:59:41.946871 8018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45219 every 8 connection(s)
I20260620 02:59:41.948180 7942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:59:41.949697 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 7942
I20260620 02:59:41.949924 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 02:59:41.950743 388 external_mini_cluster.cc:1428] Setting key 90322c2d22d849f6fbce729897253260
I20260620 02:59:41.952711 8019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:41.963359 8019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d: Bootstrap starting.
Jun 20 02:59:41 dist-test-slave-f1j0 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:59:41.966719 8019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:41.967748 8019 log.cc:824] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:41.971011 8019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d: No bootstrap required, opened a new log
I20260620 02:59:41.971941 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:41.952791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46624 (local address 127.0.97.62:45219)
0620 02:59:41.953500 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:41.953511 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:41.953551 (+ 40us) server_negotiation.cc:407] Connection header received
0620 02:59:41.955178 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:41.955198 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:41.955597 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:41.955985 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:41.957362 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:41.958740 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:41.959929 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:41.960599 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:41.960846 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:41.964672 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:41.964681 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:41.964705 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:41.964742 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:41.967776 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:41.968635 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:41.968647 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:41.968657 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:41.968946 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:41.969353 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:41.969356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:41.969359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:41.969895 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:41.970086 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:41.970579 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 02:59:41.970916 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":331,"threads_started":1}
I20260620 02:59:41.978252 8019 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8bb4209e67c43ddb0af4b1500f7d16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45219 } }
I20260620 02:59:41.978652 8019 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:41.978755 8019 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8bb4209e67c43ddb0af4b1500f7d16d, State: Initialized, Role: FOLLOWER
I20260620 02:59:41.979287 8019 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8bb4209e67c43ddb0af4b1500f7d16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45219 } }
I20260620 02:59:41.979521 8019 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:59:41.979656 8019 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:59:41.979828 8019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:41.981963 8019 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8bb4209e67c43ddb0af4b1500f7d16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45219 } }
I20260620 02:59:41.982486 8019 leader_election.cc:302] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8bb4209e67c43ddb0af4b1500f7d16d; no voters:
I20260620 02:59:41.983022 8019 leader_election.cc:288] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:59:41.983142 8024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:41.983862 8024 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d [term 1 LEADER]: Becoming Leader. State: Replica: e8bb4209e67c43ddb0af4b1500f7d16d, State: Running, Role: LEADER
I20260620 02:59:41.984409 8024 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8bb4209e67c43ddb0af4b1500f7d16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45219 } }
I20260620 02:59:41.985074 8019 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:59:41.987429 8026 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8bb4209e67c43ddb0af4b1500f7d16d. Latest consensus state: current_term: 1 leader_uuid: "e8bb4209e67c43ddb0af4b1500f7d16d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8bb4209e67c43ddb0af4b1500f7d16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45219 } } }
I20260620 02:59:41.987967 8026 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:41.987425 8025 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8bb4209e67c43ddb0af4b1500f7d16d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8bb4209e67c43ddb0af4b1500f7d16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45219 } } }
I20260620 02:59:41.988481 8025 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:41.994076 8030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:59:41.999254 8030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:59:42.008445 8030 catalog_manager.cc:1347] Generated new cluster ID: 2804d3d799f146b3adefcb2a4f8e55cd
I20260620 02:59:42.008582 8030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:59:42.031051 8030 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:59:42.032351 8030 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:59:42.046166 8030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d: Generated new TSK 0
I20260620 02:59:42.047005 8030 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 02:59:42.105196 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45219
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:42.459326 8045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:42.459616 8045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:42.459714 8045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:42.467128 8045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:42.467242 8045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:42.467330 8045 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:59:42.467410 8045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:59:42.477208 8045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:42.478580 8045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:42.479832 8045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:42.493045 8053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:42.493137 8050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:42.493119 8051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:42.493839 8045 server_base.cc:1034] running on GCE node
I20260620 02:59:42.494532 8045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:42.495546 8045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:42.496755 8045 hybrid_clock.cc:648] HybridClock initialized: now 1781924382496658 us; error 71 us; skew 500 ppm
I20260620 02:59:42.497071 8045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:42 dist-test-slave-f1j0 krb5kdc[7929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924382, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:42.500216 8045 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 02:59:42.501672 8045 webserver.cc:460] Webserver started at http://127.0.97.1:39359/ using document root <none> and password file <none>
I20260620 02:59:42.502326 8045 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:42.502413 8045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:42.502663 8045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:42.505455 8045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "16a51aaa9abc45c99545598d6f5f04c0"
format_stamp: "Formatted at 2026-06-20 02:59:42 on dist-test-slave-f1j0"
server_key: "460e3b8ee29f63137d91bf147145590b"
server_key_iv: "169da74b669e05bd22ef7f06da8225e7"
server_key_version: "encryptionkey@0"
I20260620 02:59:42.506263 8045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "16a51aaa9abc45c99545598d6f5f04c0"
format_stamp: "Formatted at 2026-06-20 02:59:42 on dist-test-slave-f1j0"
server_key: "460e3b8ee29f63137d91bf147145590b"
server_key_iv: "169da74b669e05bd22ef7f06da8225e7"
server_key_version: "encryptionkey@0"
I20260620 02:59:42.511348 8045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:59:42.514631 8060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:42.516315 8045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:59:42.516491 8045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "16a51aaa9abc45c99545598d6f5f04c0"
format_stamp: "Formatted at 2026-06-20 02:59:42 on dist-test-slave-f1j0"
server_key: "460e3b8ee29f63137d91bf147145590b"
server_key_iv: "169da74b669e05bd22ef7f06da8225e7"
server_key_version: "encryptionkey@0"
I20260620 02:59:42.516763 8045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:42.529238 8045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:42.533355 8045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:42.533715 8045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:42.536343 8045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:42.536450 8045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:42.536633 8045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:42.536746 8045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 02:59:42 dist-test-slave-f1j0 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924382, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:59:42.571480 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:42.537603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45219 (local address 127.0.97.1:45863)
0620 02:59:42.547804 (+ 10201us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:42.547856 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:42.550228 (+ 2372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:42.551064 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:42.551073 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:42.551600 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:42.552190 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:42.552207 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:42.553304 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:42.553310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:42.555525 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:42.555548 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:42.556695 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:42.556716 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:42.557036 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:42.557995 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:42.558031 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:42.561026 (+ 2995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:42.565179 (+ 4153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:42.565194 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:42.565219 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:42.566054 (+ 835us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:42.566503 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:42.566507 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:42.566509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:42.566699 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:42.567533 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:42.567549 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:42.568157 (+ 608us) client_negotiation.cc:769] Sending connection context
0620 02:59:42.568566 (+ 409us) client_negotiation.cc:240] Negotiation successful
0620 02:59:42.569356 (+ 790us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9904,"thread_start_us":8383,"threads_started":1}
I20260620 02:59:42.572105 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:42.536856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45863 (local address 127.0.97.62:45219)
0620 02:59:42.545816 (+ 8960us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:42.545826 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:42.548004 (+ 2178us) server_negotiation.cc:407] Connection header received
0620 02:59:42.550590 (+ 2586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:42.550595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:42.550684 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:42.550826 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:42.552358 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:42.553078 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:42.555698 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:42.556360 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:42.556522 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:42.561281 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:42.561292 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:42.561300 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:42.561354 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:42.564996 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:42.566183 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:42.566188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:42.566195 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:42.566375 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:42.566836 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:42.566844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:42.566850 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:42.567285 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:42.567448 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:42.571522 (+ 4074us) server_negotiation.cc:299] Negotiation successful
0620 02:59:42.571763 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8748,"thread_start_us":8620,"threads_started":1}
I20260620 02:59:42.590901 8045 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36653
I20260620 02:59:42.590948 8180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36653 every 8 connection(s)
I20260620 02:59:42.592270 8045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:59:42.594807 8181 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45219
I20260620 02:59:42.595019 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:42.579065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46371 (local address 127.0.97.62:45219)
0620 02:59:42.579267 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:42.579273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:42.579934 (+ 661us) server_negotiation.cc:407] Connection header received
0620 02:59:42.580123 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:42.580130 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:42.580221 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:42.580339 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:42.581660 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:42.582833 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:42.585400 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:42.586110 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:42.586252 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:42.588965 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:42.588970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:42.588974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:42.589003 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:42.591974 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:42.592697 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:42.592704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:42.592709 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:42.592881 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:42.593330 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:42.593338 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:42.593343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:42.593728 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:42.593926 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:42.594383 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 02:59:42.594601 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:59:42.595132 8181 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:42.596232 8181 heartbeater.cc:511] Master 127.0.97.62:45219 requested a full tablet report, sending...
I20260620 02:59:42.595139 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:42.579120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45219 (local address 127.0.97.1:46371)
0620 02:59:42.579830 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:42.579846 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:42.580004 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:42.580393 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:42.580401 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:42.581048 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:42.581464 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:42.581478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:42.582993 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:42.582998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:42.585228 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:42.585246 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:42.586329 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:42.586335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:42.586512 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:42.587343 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:42.587376 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:42.588714 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:42.592109 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:42.592115 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:42.592119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:42.592581 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:42.593022 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:42.593027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:42.593030 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:42.593181 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:42.593890 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:42.593897 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:42.594111 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 02:59:42.594267 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 02:59:42.594610 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":456,"threads_started":1}
I20260620 02:59:42.599263 7974 ts_manager.cc:194] Registered new tserver with Master: 16a51aaa9abc45c99545598d6f5f04c0 (127.0.97.1:36653)
I20260620 02:59:42.599439 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 8045
I20260620 02:59:42.599642 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:59:42.600412 388 external_mini_cluster.cc:1428] Setting key 6c2411a4c8b5493957bb953e5b6f7321
I20260620 02:59:42.602739 7974 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45863
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 02:59:42.648559 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45219
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:42.967594 8185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:42.967868 8185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:42.968050 8185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:42.975108 8185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:42.975252 8185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:42.975400 8185 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:59:42.975531 8185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:59:42.984943 8185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:42.986337 8185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:42.987705 8185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:43.000918 8191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:43.001060 8190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:43.000901 8193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:43.001504 8185 server_base.cc:1034] running on GCE node
I20260620 02:59:43.002204 8185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:43.003068 8185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:43.004294 8185 hybrid_clock.cc:648] HybridClock initialized: now 1781924383004225 us; error 28 us; skew 500 ppm
I20260620 02:59:43.004549 8185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:43 dist-test-slave-f1j0 krb5kdc[7929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:43.007647 8185 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 02:59:43.009074 8185 webserver.cc:460] Webserver started at http://127.0.97.2:44541/ using document root <none> and password file <none>
I20260620 02:59:43.009795 8185 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:43.009881 8185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:43.010141 8185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:43.012884 8185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "303c83a362da47ad97718d60bcc96b3a"
format_stamp: "Formatted at 2026-06-20 02:59:43 on dist-test-slave-f1j0"
server_key: "a5e5565db174918ce76708ae75f1fdb5"
server_key_iv: "6ada28f83042e05a30a05c6aab386d5e"
server_key_version: "encryptionkey@0"
I20260620 02:59:43.013651 8185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "303c83a362da47ad97718d60bcc96b3a"
format_stamp: "Formatted at 2026-06-20 02:59:43 on dist-test-slave-f1j0"
server_key: "a5e5565db174918ce76708ae75f1fdb5"
server_key_iv: "6ada28f83042e05a30a05c6aab386d5e"
server_key_version: "encryptionkey@0"
I20260620 02:59:43.018720 8185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:59:43.022217 8200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:43.023571 8185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:59:43.023697 8185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "303c83a362da47ad97718d60bcc96b3a"
format_stamp: "Formatted at 2026-06-20 02:59:43 on dist-test-slave-f1j0"
server_key: "a5e5565db174918ce76708ae75f1fdb5"
server_key_iv: "6ada28f83042e05a30a05c6aab386d5e"
server_key_version: "encryptionkey@0"
I20260620 02:59:43.023871 8185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:43.034713 8185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:43.038262 8185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:43.038585 8185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:43.041031 8185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:43.041136 8185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:43.041250 8185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:43.041333 8185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:59:43 dist-test-slave-f1j0 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:59:43.064741 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.041798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43469 (local address 127.0.97.62:45219)
0620 02:59:43.042065 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:43.042069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:43.043290 (+ 1221us) server_negotiation.cc:407] Connection header received
0620 02:59:43.045470 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:43.045475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:43.045588 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:43.045716 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:43.048144 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.049444 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.051732 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.052344 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.052478 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.057266 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:43.057272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:43.057275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:43.057328 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:43.061107 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:43.062168 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:43.062173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:43.062177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:43.062303 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:43.062884 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:43.062889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:43.062892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:43.063295 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:43.063469 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:43.064148 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 02:59:43.064378 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260620 02:59:43.066135 8209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.042395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45219 (local address 127.0.97.2:43469)
0620 02:59:43.043128 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:43.043176 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:43.045127 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:43.046082 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:43.046093 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:43.047018 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:43.047904 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:43.047937 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.049642 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.049659 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:43.051578 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:43.051589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.052585 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.052594 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:43.052822 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.053671 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:43.053692 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:43.057002 (+ 3310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:43.061285 (+ 4283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:43.061294 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:43.061316 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:43.062030 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:43.062462 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:43.062468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:43.062473 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:43.062761 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:43.063435 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:43.063445 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:43.063805 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 02:59:43.064060 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 02:59:43.064583 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":313,"threads_started":1}
I20260620 02:59:43.090200 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.074059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46443 (local address 127.0.97.62:45219)
0620 02:59:43.074248 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:43.074254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:43.074939 (+ 685us) server_negotiation.cc:407] Connection header received
0620 02:59:43.075227 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:43.075236 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:43.075340 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:43.075492 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:43.077110 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.077898 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.080512 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.081027 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.081163 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.083743 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:43.083753 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:43.083760 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:43.083820 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:43.087357 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:43.088212 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:43.088220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:43.088225 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:43.088346 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:43.088706 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:43.088713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:43.088716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:43.089025 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:43.089178 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:43.089617 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 02:59:43.089848 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:59:43.090516 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.074121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45219 (local address 127.0.97.2:46443)
0620 02:59:43.074794 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:43.074825 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:43.074996 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:43.075739 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:43.075747 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:43.076464 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:43.076946 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:43.076962 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.078041 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.078050 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:43.080345 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:43.080366 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.081360 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.081367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:43.081636 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.082394 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:43.082410 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:43.083528 (+ 1118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:43.087496 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:43.087501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:43.087505 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:43.088087 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:43.088461 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:43.088466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:43.088469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:43.088598 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:43.089139 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:43.089152 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:43.089357 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 02:59:43.089496 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 02:59:43.089948 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":367,"threads_started":1}
I20260620 02:59:43.090549 8185 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34783
I20260620 02:59:43.090591 8319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34783 every 8 connection(s)
I20260620 02:59:43.092057 8185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:59:43.094671 8320 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45219
I20260620 02:59:43.094903 8320 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:43.095554 8320 heartbeater.cc:511] Master 127.0.97.62:45219 requested a full tablet report, sending...
I20260620 02:59:43.096814 7974 ts_manager.cc:194] Registered new tserver with Master: 303c83a362da47ad97718d60bcc96b3a (127.0.97.2:34783)
I20260620 02:59:43.097633 7974 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43469
I20260620 02:59:43.099792 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 8185
I20260620 02:59:43.099982 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:59:43.100389 388 external_mini_cluster.cc:1428] Setting key 8fcf7c779b5ebba6cd4d22845fdbd79f
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 02:59:43.146792 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45219
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:43.470956 8324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:43.471298 8324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:43.471479 8324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:43.479012 8324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:43.479154 8324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:43.479307 8324 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:59:43.479461 8324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:59:43.489477 8324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:43.490856 8324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:43.492202 8324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:43.505836 8329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:43.505865 8332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:43.505816 8330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:43.506774 8324 server_base.cc:1034] running on GCE node
I20260620 02:59:43.507433 8324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:43.508383 8324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:43.509684 8324 hybrid_clock.cc:648] HybridClock initialized: now 1781924383509607 us; error 60 us; skew 500 ppm
I20260620 02:59:43.509981 8324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:43 dist-test-slave-f1j0 krb5kdc[7929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:43.512974 8324 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 02:59:43.514442 8324 webserver.cc:460] Webserver started at http://127.0.97.3:34291/ using document root <none> and password file <none>
I20260620 02:59:43.515118 8324 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:43.515247 8324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:43.515558 8324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:43.518417 8324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "ce0762f2fb2a41b4b7ae13351fdead66"
format_stamp: "Formatted at 2026-06-20 02:59:43 on dist-test-slave-f1j0"
server_key: "c9e46684c6b0febaee0f9ad2b5947bce"
server_key_iv: "aa77e333242f2d77e35834f190003fd8"
server_key_version: "encryptionkey@0"
I20260620 02:59:43.519209 8324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ce0762f2fb2a41b4b7ae13351fdead66"
format_stamp: "Formatted at 2026-06-20 02:59:43 on dist-test-slave-f1j0"
server_key: "c9e46684c6b0febaee0f9ad2b5947bce"
server_key_iv: "aa77e333242f2d77e35834f190003fd8"
server_key_version: "encryptionkey@0"
I20260620 02:59:43.524621 8324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 02:59:43.528010 8339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:43.529364 8324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:59:43.529526 8324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "ce0762f2fb2a41b4b7ae13351fdead66"
format_stamp: "Formatted at 2026-06-20 02:59:43 on dist-test-slave-f1j0"
server_key: "c9e46684c6b0febaee0f9ad2b5947bce"
server_key_iv: "aa77e333242f2d77e35834f190003fd8"
server_key_version: "encryptionkey@0"
I20260620 02:59:43.529793 8324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:43.541432 8324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:43.545193 8324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:43.545465 8324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:43.548224 8324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:43.548329 8324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:43.548530 8324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:43.548647 8324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:59:43 dist-test-slave-f1j0 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:59:43.570394 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.548673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38253 (local address 127.0.97.62:45219)
0620 02:59:43.548870 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:43.548876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:43.550233 (+ 1357us) server_negotiation.cc:407] Connection header received
0620 02:59:43.552187 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:43.552192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:43.552285 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:43.552397 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:43.554526 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.555609 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.558159 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.558827 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.558995 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.562792 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:43.562798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:43.562804 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:43.562852 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:43.566389 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:43.567451 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:43.567456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:43.567460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:43.567587 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:43.568137 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:43.568141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:43.568146 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:43.568536 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:43.568688 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:43.569894 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0620 02:59:43.570144 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:59:43.572517 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.549192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45219 (local address 127.0.97.3:38253)
0620 02:59:43.550054 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:43.550102 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:43.551904 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:43.552643 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:43.552651 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:43.553361 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:43.554334 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:43.554368 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.555752 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.555761 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:43.557969 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:43.558003 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.559116 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.559125 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:43.559365 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.560031 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:43.560052 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:43.562497 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:43.566548 (+ 4051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:43.566556 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:43.566586 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:43.567337 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:43.567701 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:43.567706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:43.567715 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:43.568009 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:43.568710 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:43.568722 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:43.569326 (+ 604us) client_negotiation.cc:769] Sending connection context
0620 02:59:43.569731 (+ 405us) client_negotiation.cc:240] Negotiation successful
0620 02:59:43.570485 (+ 754us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":411,"threads_started":1}
I20260620 02:59:43.597949 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.580115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39371 (local address 127.0.97.62:45219)
0620 02:59:43.580321 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:43.580329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:43.580900 (+ 571us) server_negotiation.cc:407] Connection header received
0620 02:59:43.581089 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:43.581095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:43.581166 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:43.581269 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:43.582238 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.582939 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.584241 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.584682 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.584828 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.587540 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:43.587545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:43.587548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:43.587606 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:43.594744 (+ 7138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:43.595544 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:43.595551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:43.595554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:43.595702 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:43.596092 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:43.596097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:43.596101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:43.596498 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:43.596635 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:43.597269 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 02:59:43.597481 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:59:43.598253 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.579644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45219 (local address 127.0.97.3:39371)
0620 02:59:43.580802 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:43.580819 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:43.580968 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:43.581292 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:43.581296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:43.581822 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:43.582094 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:43.582103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.583082 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.583087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:43.584099 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:43.584110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.584866 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.584873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:43.585104 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.585916 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:43.585946 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:43.587216 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:43.594891 (+ 7675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:43.594897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:43.594900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:43.595423 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:43.595821 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:43.595826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:43.595830 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:43.595964 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:43.596633 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:43.596642 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:43.596919 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 02:59:43.597123 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 02:59:43.597594 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"thread_start_us":860,"threads_started":1}
I20260620 02:59:43.604779 8324 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42707
I20260620 02:59:43.604806 8458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42707 every 8 connection(s)
I20260620 02:59:43.606091 8324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:59:43.606344 8181 heartbeater.cc:503] Master 127.0.97.62:45219 was elected leader, sending a full tablet report...
I20260620 02:59:43.608202 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 8324
I20260620 02:59:43.608402 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:59:43.608383 8459 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45219
I20260620 02:59:43.608714 8459 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:43.608772 388 external_mini_cluster.cc:1428] Setting key e3ce4caeec9ad490c425b0f89fbe51e4
I20260620 02:59:43.609879 8459 heartbeater.cc:511] Master 127.0.97.62:45219 requested a full tablet report, sending...
I20260620 02:59:43.611730 7974 ts_manager.cc:194] Registered new tserver with Master: ce0762f2fb2a41b4b7ae13351fdead66 (127.0.97.3:42707)
I20260620 02:59:43.612536 7974 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38253
I20260620 02:59:43.623847 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:59:43 dist-test-slave-f1j0 krb5kdc[7929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924383, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:59:43 dist-test-slave-f1j0 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:43 dist-test-slave-f1j0 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:59:43.676605 8069 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:43.666086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40702 (local address 127.0.97.62:45219)
0620 02:59:43.666298 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:43.666307 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:43.666696 (+ 389us) server_negotiation.cc:407] Connection header received
0620 02:59:43.666848 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:43.666851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:43.666922 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:43.667018 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:43.668494 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.669193 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.670504 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.670908 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.671013 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.676390 (+ 5377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40702: BlockingRecv error: recv got EOF from 127.0.0.1:40702 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 02:59:43 dist-test-slave-f1j0 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924383, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:59:43.692052 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.679491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40710 (local address 127.0.97.62:45219)
0620 02:59:43.679668 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:43.679674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:43.680074 (+ 400us) server_negotiation.cc:407] Connection header received
0620 02:59:43.680198 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:43.680202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:43.680282 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:43.680386 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:43.681390 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.682144 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.683311 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.683699 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.683789 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.687328 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:43.687333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:43.687337 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:43.687367 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:43.689969 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:43.690531 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:43.690535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:43.690537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:43.690644 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:43.690937 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:43.690940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:43.690942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:43.691171 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:43.691378 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:43.691558 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 02:59:43.691715 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 02:59:43.696033 7974 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:59:43.702921 7974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:59:43.727270 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.715481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36653 (local address 127.0.0.1:57628)
0620 02:59:43.717079 (+ 1598us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:43.717132 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:43.717300 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:43.719782 (+ 2482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:43.719790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:43.719822 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:43.720250 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:43.720261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.722416 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.722423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:43.724459 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:43.724472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.726283 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.726289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:43.726474 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.726532 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 02:59:43.726653 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 02:59:43.726848 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1328,"mutex_wait_us":1,"thread_start_us":4010,"threads_started":1}
I20260620 02:59:43.727304 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.715731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57628 (local address 127.0.97.1:36653)
0620 02:59:43.717929 (+ 2198us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:43.717938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:43.717966 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:59:43.719217 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:43.719226 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:43.719510 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:43.719777 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:43.720474 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.722253 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.724657 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.726094 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.726234 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.726462 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:43.726731 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 02:59:43.726974 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2024,"thread_start_us":1832,"threads_started":1}
I20260620 02:59:43.727988 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.715737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42707 (local address 127.0.0.1:49824)
0620 02:59:43.718270 (+ 2533us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:43.718290 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:43.718424 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:43.720223 (+ 1799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:43.720227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:43.720249 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:43.720659 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:43.720669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.722770 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.722775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:43.725076 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:43.725089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.727078 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.727087 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:43.727323 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.727366 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 02:59:43.727442 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 02:59:43.727585 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2379,"mutex_wait_us":26,"thread_start_us":918,"threads_started":1}
I20260620 02:59:43.728684 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.716329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.0.97.3:42707)
0620 02:59:43.719563 (+ 3234us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:43.719572 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:43.719601 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:59:43.719722 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:43.719730 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:43.720010 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:43.720858 (+ 848us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:43.721210 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.722649 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.725247 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.726840 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.727697 (+ 857us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.727933 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:43.728139 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 02:59:43.728314 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3056,"thread_start_us":2832,"threads_started":1}
I20260620 02:59:43.729110 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.717450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34783 (local address 127.0.0.1:52216)
0620 02:59:43.719830 (+ 2380us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:43.719849 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:43.720038 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:43.720633 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:43.720640 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:43.720664 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:43.721045 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:43.721057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.723151 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.723156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:43.725354 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:43.725363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.728367 (+ 3004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.728375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:43.728537 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.728565 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 02:59:43.728653 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 02:59:43.728783 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2256,"mutex_wait_us":58,"thread_start_us":1352,"threads_started":1}
I20260620 02:59:43.729734 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.717961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52216 (local address 127.0.97.2:34783)
0620 02:59:43.718947 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:43.718959 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:43.719938 (+ 979us) server_negotiation.cc:407] Connection header received
0620 02:59:43.720237 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:43.720243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:43.720432 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:43.720633 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:43.721221 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.723012 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.725489 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.727925 (+ 2436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.728045 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.728225 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:43.729334 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0620 02:59:43.729466 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":658,"threads_started":1}
I20260620 02:59:43.731879 8111 tablet_service.cc:1467] Processing CreateTablet for tablet e984fe41bfca42ddacb6714e4b745bb2 (DEFAULT_TABLE table=test-table [id=e3b81f02dbaa4cd0a07965bd0162eb12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:43.732340 8389 tablet_service.cc:1467] Processing CreateTablet for tablet e984fe41bfca42ddacb6714e4b745bb2 (DEFAULT_TABLE table=test-table [id=e3b81f02dbaa4cd0a07965bd0162eb12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:43.733521 8111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e984fe41bfca42ddacb6714e4b745bb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:43.733829 8389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e984fe41bfca42ddacb6714e4b745bb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:43.733551 8250 tablet_service.cc:1467] Processing CreateTablet for tablet e984fe41bfca42ddacb6714e4b745bb2 (DEFAULT_TABLE table=test-table [id=e3b81f02dbaa4cd0a07965bd0162eb12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:43.734791 8250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e984fe41bfca42ddacb6714e4b745bb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:43.746048 8483 tablet_bootstrap.cc:492] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a: Bootstrap starting.
I20260620 02:59:43.749158 8484 tablet_bootstrap.cc:492] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0: Bootstrap starting.
I20260620 02:59:43.749512 8483 tablet_bootstrap.cc:654] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:43.750941 8483 log.cc:824] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:43.751084 8485 tablet_bootstrap.cc:492] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66: Bootstrap starting.
I20260620 02:59:43.752564 8484 tablet_bootstrap.cc:654] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:43.753831 8484 log.cc:824] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:43.754635 8483 tablet_bootstrap.cc:492] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a: No bootstrap required, opened a new log
I20260620 02:59:43.754884 8485 tablet_bootstrap.cc:654] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:43.754899 8483 ts_tablet_manager.cc:1397] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260620 02:59:43.755893 8485 log.cc:824] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:43.756736 8484 tablet_bootstrap.cc:492] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0: No bootstrap required, opened a new log
I20260620 02:59:43.756984 8484 ts_tablet_manager.cc:1397] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260620 02:59:43.758460 8485 tablet_bootstrap.cc:492] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66: No bootstrap required, opened a new log
I20260620 02:59:43.758718 8485 ts_tablet_manager.cc:1397] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 02:59:43.765352 8483 raft_consensus.cc:348] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } }
I20260620 02:59:43.765815 8483 raft_consensus.cc:374] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:43.765938 8483 raft_consensus.cc:729] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303c83a362da47ad97718d60bcc96b3a, State: Initialized, Role: FOLLOWER
I20260620 02:59:43.766563 8483 consensus_queue.cc:260] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } }
I20260620 02:59:43.766659 8485 raft_consensus.cc:348] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } }
I20260620 02:59:43.767057 8485 raft_consensus.cc:374] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:43.767239 8485 raft_consensus.cc:729] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0762f2fb2a41b4b7ae13351fdead66, State: Initialized, Role: FOLLOWER
I20260620 02:59:43.767201 8484 raft_consensus.cc:348] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } }
I20260620 02:59:43.767695 8320 heartbeater.cc:503] Master 127.0.97.62:45219 was elected leader, sending a full tablet report...
I20260620 02:59:43.768021 8485 consensus_queue.cc:260] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } }
I20260620 02:59:43.768411 8483 ts_tablet_manager.cc:1428] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20260620 02:59:43.768769 8484 raft_consensus.cc:374] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:43.768909 8484 raft_consensus.cc:729] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a51aaa9abc45c99545598d6f5f04c0, State: Initialized, Role: FOLLOWER
I20260620 02:59:43.769587 8459 heartbeater.cc:503] Master 127.0.97.62:45219 was elected leader, sending a full tablet report...
I20260620 02:59:43.769593 8484 consensus_queue.cc:260] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } }
I20260620 02:59:43.770220 8485 ts_tablet_manager.cc:1428] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260620 02:59:43.771202 8484 ts_tablet_manager.cc:1428] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260620 02:59:43.845371 8321 tablet.cc:2369] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:59:43.857311 8182 tablet.cc:2369] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:59:43.860803 8490 raft_consensus.cc:482] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:59:43.861093 8490 raft_consensus.cc:504] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } }
I20260620 02:59:43.862784 8490 leader_election.cc:288] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16a51aaa9abc45c99545598d6f5f04c0 (127.0.97.1:36653), 303c83a362da47ad97718d60bcc96b3a (127.0.97.2:34783)
W20260620 02:59:43.863847 8460 tablet.cc:2369] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:59:43.869196 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.863026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34783 (local address 127.0.97.3:53981)
0620 02:59:43.863280 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:43.863302 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:43.863431 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:43.863977 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:43.863985 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:43.864016 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:43.864294 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:43.864306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.865599 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.865606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:43.867329 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:43.867364 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.868606 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.868612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:43.868742 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.868765 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 02:59:43.868834 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 02:59:43.868961 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":2,"spinlock_wait_cycles":49792}
I20260620 02:59:43.869266 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.863388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53981 (local address 127.0.97.2:34783)
0620 02:59:43.863614 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:43.863622 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:43.863644 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:59:43.863724 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:43.863729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:43.863819 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:43.863928 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:43.864456 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.865451 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.867557 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.868441 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.868571 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.868673 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:43.868884 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 02:59:43.869009 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:59:43.870393 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.863196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36653 (local address 127.0.97.3:35221)
0620 02:59:43.863938 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:43.863953 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:43.864135 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:43.864651 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:43.864654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:43.864676 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:43.865024 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:43.865034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.866300 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.866305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:43.868253 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:43.868265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.869860 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.869865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:43.869973 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.869995 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 02:59:43.870066 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 02:59:43.870143 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"spinlock_wait_cycles":896,"thread_start_us":924,"threads_started":1}
I20260620 02:59:43.870267 8270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e984fe41bfca42ddacb6714e4b745bb2" candidate_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303c83a362da47ad97718d60bcc96b3a" is_pre_election: true
I20260620 02:59:43.870471 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.863207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35221 (local address 127.0.97.1:36653)
0620 02:59:43.863409 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:43.863418 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:43.864026 (+ 608us) server_negotiation.cc:407] Connection header received
0620 02:59:43.864312 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:43.864318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:43.864410 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:43.864469 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:43.865181 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.866133 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.868430 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.869654 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.869812 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.869914 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:43.870115 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 02:59:43.870233 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:59:43.870818 8270 raft_consensus.cc:2454] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce0762f2fb2a41b4b7ae13351fdead66 in term 0.
I20260620 02:59:43.871500 8343 leader_election.cc:302] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 303c83a362da47ad97718d60bcc96b3a, ce0762f2fb2a41b4b7ae13351fdead66; no voters:
I20260620 02:59:43.871470 8131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e984fe41bfca42ddacb6714e4b745bb2" candidate_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a51aaa9abc45c99545598d6f5f04c0" is_pre_election: true
I20260620 02:59:43.871914 8490 raft_consensus.cc:2781] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:59:43.872048 8490 raft_consensus.cc:482] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:59:43.871989 8131 raft_consensus.cc:2454] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce0762f2fb2a41b4b7ae13351fdead66 in term 0.
I20260620 02:59:43.872167 8490 raft_consensus.cc:3037] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:43.874504 8490 raft_consensus.cc:504] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } }
I20260620 02:59:43.875159 8490 leader_election.cc:288] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [CANDIDATE]: Term 1 election: Requested vote from peers 16a51aaa9abc45c99545598d6f5f04c0 (127.0.97.1:36653), 303c83a362da47ad97718d60bcc96b3a (127.0.97.2:34783)
I20260620 02:59:43.875546 8131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e984fe41bfca42ddacb6714e4b745bb2" candidate_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a51aaa9abc45c99545598d6f5f04c0"
I20260620 02:59:43.875770 8131 raft_consensus.cc:3037] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:43.875720 8270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e984fe41bfca42ddacb6714e4b745bb2" candidate_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303c83a362da47ad97718d60bcc96b3a"
I20260620 02:59:43.875937 8270 raft_consensus.cc:3037] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:43.878222 8131 raft_consensus.cc:2454] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce0762f2fb2a41b4b7ae13351fdead66 in term 1.
I20260620 02:59:43.878495 8270 raft_consensus.cc:2454] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce0762f2fb2a41b4b7ae13351fdead66 in term 1.
I20260620 02:59:43.878611 8341 leader_election.cc:302] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a51aaa9abc45c99545598d6f5f04c0, ce0762f2fb2a41b4b7ae13351fdead66; no voters:
I20260620 02:59:43.878954 8490 raft_consensus.cc:2781] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:43.879303 8490 raft_consensus.cc:686] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [term 1 LEADER]: Becoming Leader. State: Replica: ce0762f2fb2a41b4b7ae13351fdead66, State: Running, Role: LEADER
I20260620 02:59:43.879779 8490 consensus_queue.cc:237] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } }
I20260620 02:59:43.885239 7972 catalog_manager.cc:5495] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce0762f2fb2a41b4b7ae13351fdead66 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260620 02:59:43.915879 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:43.910762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49828 (local address 127.0.97.3:42707)
0620 02:59:43.910961 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:43.910969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:43.910988 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:59:43.911168 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:43.911171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:43.911253 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:43.911407 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:59:43.911739 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.912444 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.913821 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:43.914223 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:43.914358 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:43.914779 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:59:43.915313 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:59:43.915466 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:43.915541 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 02:59:43.915617 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:59:43.924402 8131 raft_consensus.cc:1261] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 FOLLOWER]: Refusing update from remote peer ce0762f2fb2a41b4b7ae13351fdead66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:43.924400 8270 raft_consensus.cc:1261] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a [term 1 FOLLOWER]: Refusing update from remote peer ce0762f2fb2a41b4b7ae13351fdead66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:43.925303 8493 consensus_queue.cc:1035] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:43.925693 8490 consensus_queue.cc:1035] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:43.936484 8496 mvcc.cc:205] Tried to move back new op lower bound from 7298762276544548864 to 7298762276375293952. Current Snapshot: MvccSnapshot[applied={T|T < 7298762276544548864}]
I20260620 02:59:43.939508 8497 mvcc.cc:205] Tried to move back new op lower bound from 7298762276544548864 to 7298762276375293952. Current Snapshot: MvccSnapshot[applied={T|T < 7298762276544548864}]
I20260620 02:59:43.950780 7974 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:43.951089 7974 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:43.954990 7974 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d: Sending DeleteTablet for 3 replicas of tablet e984fe41bfca42ddacb6714e4b745bb2
I20260620 02:59:43.956048 8389 tablet_service.cc:1514] Processing DeleteTablet for tablet e984fe41bfca42ddacb6714e4b745bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:43 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49824
I20260620 02:59:43.956048 8111 tablet_service.cc:1514] Processing DeleteTablet for tablet e984fe41bfca42ddacb6714e4b745bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:43 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57628
I20260620 02:59:43.956871 8506 tablet_replica.cc:330] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0: stopping tablet replica
I20260620 02:59:43.957278 8506 raft_consensus.cc:2229] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:43.957206 8250 tablet_service.cc:1514] Processing DeleteTablet for tablet e984fe41bfca42ddacb6714e4b745bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:43 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52216
I20260620 02:59:43.957823 8506 raft_consensus.cc:2258] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:43.958485 8507 tablet_replica.cc:330] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66: stopping tablet replica
I20260620 02:59:43.958864 8507 raft_consensus.cc:2229] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:59:43.959405 8507 raft_consensus.cc:2258] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:43.960088 7974 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:59:43.960920 7974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:59:43.961323 8507 ts_tablet_manager.cc:1905] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:43.968153 8507 ts_tablet_manager.cc:1918] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:59:43.968389 8507 log.cc:1196] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/e984fe41bfca42ddacb6714e4b745bb2
I20260620 02:59:43.968920 8507 ts_tablet_manager.cc:1939] T e984fe41bfca42ddacb6714e4b745bb2 P ce0762f2fb2a41b4b7ae13351fdead66: Deleting consensus metadata
I20260620 02:59:43.970078 8111 tablet_service.cc:1467] Processing CreateTablet for tablet 5efd979abbf84faba719beeb67a49c99 (DEFAULT_TABLE table=test-table [id=b48d01351d2646e78272c6c71b176a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:43.970065 8249 tablet_service.cc:1467] Processing CreateTablet for tablet 5efd979abbf84faba719beeb67a49c99 (DEFAULT_TABLE table=test-table [id=b48d01351d2646e78272c6c71b176a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:43.970515 8111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efd979abbf84faba719beeb67a49c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:43.970604 8249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efd979abbf84faba719beeb67a49c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:43.971020 8506 ts_tablet_manager.cc:1905] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:43.974614 7961 catalog_manager.cc:4841] TS ce0762f2fb2a41b4b7ae13351fdead66 (127.0.97.3:42707): tablet e984fe41bfca42ddacb6714e4b745bb2 (table test-table [id=e3b81f02dbaa4cd0a07965bd0162eb12]) successfully deleted
I20260620 02:59:43.975232 8389 tablet_service.cc:1467] Processing CreateTablet for tablet 5efd979abbf84faba719beeb67a49c99 (DEFAULT_TABLE table=test-table [id=b48d01351d2646e78272c6c71b176a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:43.975715 8389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efd979abbf84faba719beeb67a49c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:43.975960 8483 tablet_bootstrap.cc:492] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a: Bootstrap starting.
I20260620 02:59:43.977555 8483 tablet_bootstrap.cc:654] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:43.977743 8506 ts_tablet_manager.cc:1918] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:59:43.977896 8506 log.cc:1196] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/e984fe41bfca42ddacb6714e4b745bb2
I20260620 02:59:43.978459 8506 ts_tablet_manager.cc:1939] T e984fe41bfca42ddacb6714e4b745bb2 P 16a51aaa9abc45c99545598d6f5f04c0: Deleting consensus metadata
I20260620 02:59:43.980144 7959 catalog_manager.cc:4841] TS 16a51aaa9abc45c99545598d6f5f04c0 (127.0.97.1:36653): tablet e984fe41bfca42ddacb6714e4b745bb2 (table test-table [id=e3b81f02dbaa4cd0a07965bd0162eb12]) successfully deleted
I20260620 02:59:43.982242 8484 tablet_bootstrap.cc:492] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0: Bootstrap starting.
I20260620 02:59:43.983863 8484 tablet_bootstrap.cc:654] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:43.984292 8485 tablet_bootstrap.cc:492] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66: Bootstrap starting.
I20260620 02:59:43.985488 8508 tablet_replica.cc:330] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a: stopping tablet replica
I20260620 02:59:43.986163 8485 tablet_bootstrap.cc:654] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:43.986639 8508 raft_consensus.cc:2229] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:43.987866 8508 raft_consensus.cc:2258] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:43.988445 8485 tablet_bootstrap.cc:492] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66: No bootstrap required, opened a new log
I20260620 02:59:43.988587 8485 ts_tablet_manager.cc:1397] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 02:59:43.988842 8484 tablet_bootstrap.cc:492] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0: No bootstrap required, opened a new log
I20260620 02:59:43.988986 8484 ts_tablet_manager.cc:1397] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260620 02:59:43.989405 8485 raft_consensus.cc:348] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:43.989665 8485 raft_consensus.cc:374] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:43.989773 8485 raft_consensus.cc:729] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0762f2fb2a41b4b7ae13351fdead66, State: Initialized, Role: FOLLOWER
I20260620 02:59:43.989813 8484 raft_consensus.cc:348] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:43.990043 8484 raft_consensus.cc:374] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:43.990142 8484 raft_consensus.cc:729] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a51aaa9abc45c99545598d6f5f04c0, State: Initialized, Role: FOLLOWER
I20260620 02:59:43.990073 8485 consensus_queue.cc:260] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:43.990372 8484 consensus_queue.cc:260] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:43.990646 8485 ts_tablet_manager.cc:1428] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:59:43.990873 8484 ts_tablet_manager.cc:1428] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:59:43.991417 8508 ts_tablet_manager.cc:1905] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:43.992636 8483 tablet_bootstrap.cc:492] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a: No bootstrap required, opened a new log
I20260620 02:59:43.992786 8483 ts_tablet_manager.cc:1397] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20260620 02:59:43.993541 8483 raft_consensus.cc:348] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:43.993785 8483 raft_consensus.cc:374] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:43.993896 8483 raft_consensus.cc:729] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303c83a362da47ad97718d60bcc96b3a, State: Initialized, Role: FOLLOWER
I20260620 02:59:43.994112 8483 consensus_queue.cc:260] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:43.994647 8483 ts_tablet_manager.cc:1428] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:59:44.000314 8508 ts_tablet_manager.cc:1918] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:59:44.000473 8508 log.cc:1196] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/e984fe41bfca42ddacb6714e4b745bb2
I20260620 02:59:44.001026 8508 ts_tablet_manager.cc:1939] T e984fe41bfca42ddacb6714e4b745bb2 P 303c83a362da47ad97718d60bcc96b3a: Deleting consensus metadata
I20260620 02:59:44.002535 7961 catalog_manager.cc:4841] TS 303c83a362da47ad97718d60bcc96b3a (127.0.97.2:34783): tablet e984fe41bfca42ddacb6714e4b745bb2 (table test-table [id=e3b81f02dbaa4cd0a07965bd0162eb12]) successfully deleted
I20260620 02:59:44.221189 8491 raft_consensus.cc:482] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:59:44.221457 8491 raft_consensus.cc:504] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.223048 8491 leader_election.cc:288] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 303c83a362da47ad97718d60bcc96b3a (127.0.97.2:34783), ce0762f2fb2a41b4b7ae13351fdead66 (127.0.97.3:42707)
I20260620 02:59:44.231454 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:44.223252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34783 (local address 127.0.97.1:54577)
0620 02:59:44.223956 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:44.223977 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:44.224160 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:44.224835 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:44.224842 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:44.224880 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:44.225348 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:44.225363 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.226906 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.226914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:44.229001 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:44.229015 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.230628 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.230634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:44.230824 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:44.230861 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:59:44.230959 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 02:59:44.231092 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":445,"threads_started":1}
I20260620 02:59:44.231566 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:44.223468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54577 (local address 127.0.97.2:34783)
0620 02:59:44.223699 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:44.223706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:44.224052 (+ 346us) server_negotiation.cc:407] Connection header received
0620 02:59:44.224382 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:44.224391 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:44.224495 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:44.224572 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:44.225597 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.226762 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.229142 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.230407 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.230558 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:44.230662 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:44.231061 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 02:59:44.231211 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 02:59:44.232275 8270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efd979abbf84faba719beeb67a49c99" candidate_uuid: "16a51aaa9abc45c99545598d6f5f04c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303c83a362da47ad97718d60bcc96b3a" is_pre_election: true
I20260620 02:59:44.232542 8270 raft_consensus.cc:2454] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16a51aaa9abc45c99545598d6f5f04c0 in term 0.
I20260620 02:59:44.233019 8064 leader_election.cc:302] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a51aaa9abc45c99545598d6f5f04c0, 303c83a362da47ad97718d60bcc96b3a; no voters:
I20260620 02:59:44.233193 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:44.224281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35197 (local address 127.0.97.3:42707)
0620 02:59:44.224463 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:44.224469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:44.226069 (+ 1600us) server_negotiation.cc:407] Connection header received
0620 02:59:44.226251 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:44.226261 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:44.226317 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:44.226426 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:44.227105 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.228294 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.230795 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.232116 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.232258 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:44.232353 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:44.232749 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 02:59:44.232886 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:59:44.233492 8491 raft_consensus.cc:2781] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:59:44.233642 8491 raft_consensus.cc:482] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:59:44.233767 8491 raft_consensus.cc:3037] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:44.234009 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:44.224108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42707 (local address 127.0.97.1:35197)
0620 02:59:44.225968 (+ 1860us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:44.225982 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:44.226112 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:44.226442 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:44.226446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:44.226467 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:44.226884 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:44.226895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.228431 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.228436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:44.230559 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:44.230576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.232313 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.232319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:44.232570 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:44.232600 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 02:59:44.233356 (+ 756us) client_negotiation.cc:240] Negotiation successful
0620 02:59:44.233498 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1761,"thread_start_us":1596,"threads_started":1}
I20260620 02:59:44.236191 8491 raft_consensus.cc:504] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.236744 8491 leader_election.cc:288] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [CANDIDATE]: Term 1 election: Requested vote from peers 303c83a362da47ad97718d60bcc96b3a (127.0.97.2:34783), ce0762f2fb2a41b4b7ae13351fdead66 (127.0.97.3:42707)
I20260620 02:59:44.237553 8270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efd979abbf84faba719beeb67a49c99" candidate_uuid: "16a51aaa9abc45c99545598d6f5f04c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303c83a362da47ad97718d60bcc96b3a"
I20260620 02:59:44.237804 8270 raft_consensus.cc:3037] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:44.237843 8410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efd979abbf84faba719beeb67a49c99" candidate_uuid: "16a51aaa9abc45c99545598d6f5f04c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" is_pre_election: true
I20260620 02:59:44.238276 8410 raft_consensus.cc:2454] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16a51aaa9abc45c99545598d6f5f04c0 in term 0.
I20260620 02:59:44.238824 8408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efd979abbf84faba719beeb67a49c99" candidate_uuid: "16a51aaa9abc45c99545598d6f5f04c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce0762f2fb2a41b4b7ae13351fdead66"
I20260620 02:59:44.239050 8408 raft_consensus.cc:3037] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:44.240890 8270 raft_consensus.cc:2454] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16a51aaa9abc45c99545598d6f5f04c0 in term 1.
I20260620 02:59:44.241321 8064 leader_election.cc:302] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a51aaa9abc45c99545598d6f5f04c0, 303c83a362da47ad97718d60bcc96b3a; no voters:
I20260620 02:59:44.241472 8408 raft_consensus.cc:2454] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16a51aaa9abc45c99545598d6f5f04c0 in term 1.
I20260620 02:59:44.241669 8491 raft_consensus.cc:2781] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:44.242152 8491 raft_consensus.cc:686] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 LEADER]: Becoming Leader. State: Replica: 16a51aaa9abc45c99545598d6f5f04c0, State: Running, Role: LEADER
I20260620 02:59:44.242566 8491 consensus_queue.cc:237] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.246304 7972 catalog_manager.cc:5495] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16a51aaa9abc45c99545598d6f5f04c0 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "16a51aaa9abc45c99545598d6f5f04c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } health_report { overall_health: UNKNOWN } } }
Jun 20 02:59:44 dist-test-slave-f1j0 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924381, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:59:44.320629 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:44.303909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40724 (local address 127.0.97.62:45219)
0620 02:59:44.306480 (+ 2571us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:44.306484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:44.306504 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:59:44.306554 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:44.306557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:44.306609 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:44.306727 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:44.308083 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.308823 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.310662 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.311053 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.311142 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:44.315083 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:44.315088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:44.315091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:44.315118 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:44.317845 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:44.318508 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:44.318516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:44.318521 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:44.318638 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:44.319018 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:44.319023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:44.319026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:44.319274 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:44.319443 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:44.319886 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 02:59:44.320207 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":2458,"thread_start_us":2310,"threads_started":1}
I20260620 02:59:44.320632 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:44.303909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45219 (local address 127.0.0.1:40724)
0620 02:59:44.306151 (+ 2242us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:44.306176 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:44.306348 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:44.306806 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:44.306811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:44.307690 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:44.307924 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:44.307936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.308977 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.308983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:44.310523 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:44.310539 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.311231 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.311237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:44.311416 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:44.312216 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:44.312240 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:44.314881 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:44.317964 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:44.317978 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:44.317986 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:44.318390 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:44.318761 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:44.318766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:44.318770 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:44.318911 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:44.319401 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:44.319409 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:44.319568 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 02:59:44.319796 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 02:59:44.320082 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2030,"spinlock_wait_cycles":8448,"thread_start_us":1852,"threads_started":1}
I20260620 02:59:44.325727 7972 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40724:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 02:59:44.326774 7972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:59:44.333396 8111 tablet_service.cc:1467] Processing CreateTablet for tablet 58268659f1e1460c821a93569fc0d6ab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=07bd87d1c8ae47c483027db08272a9e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:59:44.333396 8250 tablet_service.cc:1467] Processing CreateTablet for tablet 58268659f1e1460c821a93569fc0d6ab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=07bd87d1c8ae47c483027db08272a9e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:59:44.333972 8250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58268659f1e1460c821a93569fc0d6ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:44.334000 8111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58268659f1e1460c821a93569fc0d6ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:44.338546 8389 tablet_service.cc:1467] Processing CreateTablet for tablet 58268659f1e1460c821a93569fc0d6ab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=07bd87d1c8ae47c483027db08272a9e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:59:44.339112 8389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58268659f1e1460c821a93569fc0d6ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:44.341006 8483 tablet_bootstrap.cc:492] T 58268659f1e1460c821a93569fc0d6ab P 303c83a362da47ad97718d60bcc96b3a: Bootstrap starting.
I20260620 02:59:44.342818 8483 tablet_bootstrap.cc:654] T 58268659f1e1460c821a93569fc0d6ab P 303c83a362da47ad97718d60bcc96b3a: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:44.345007 8483 tablet_bootstrap.cc:492] T 58268659f1e1460c821a93569fc0d6ab P 303c83a362da47ad97718d60bcc96b3a: No bootstrap required, opened a new log
I20260620 02:59:44.345184 8483 ts_tablet_manager.cc:1397] T 58268659f1e1460c821a93569fc0d6ab P 303c83a362da47ad97718d60bcc96b3a: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260620 02:59:44.345198 8485 tablet_bootstrap.cc:492] T 58268659f1e1460c821a93569fc0d6ab P ce0762f2fb2a41b4b7ae13351fdead66: Bootstrap starting.
I20260620 02:59:44.346040 8483 raft_consensus.cc:348] T 58268659f1e1460c821a93569fc0d6ab P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.346338 8483 raft_consensus.cc:374] T 58268659f1e1460c821a93569fc0d6ab P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:44.346428 8483 raft_consensus.cc:729] T 58268659f1e1460c821a93569fc0d6ab P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303c83a362da47ad97718d60bcc96b3a, State: Initialized, Role: FOLLOWER
I20260620 02:59:44.346741 8483 consensus_queue.cc:260] T 58268659f1e1460c821a93569fc0d6ab P 303c83a362da47ad97718d60bcc96b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.346947 8485 tablet_bootstrap.cc:654] T 58268659f1e1460c821a93569fc0d6ab P ce0762f2fb2a41b4b7ae13351fdead66: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:44.347203 8489 tablet_replica.cc:439] T 58268659f1e1460c821a93569fc0d6ab P 303c83a362da47ad97718d60bcc96b3a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } } }
I20260620 02:59:44.347519 8489 tablet_replica.cc:442] T 58268659f1e1460c821a93569fc0d6ab P 303c83a362da47ad97718d60bcc96b3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:59:44.350214 8484 tablet_bootstrap.cc:492] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0: Bootstrap starting.
I20260620 02:59:44.350271 8483 ts_tablet_manager.cc:1428] T 58268659f1e1460c821a93569fc0d6ab P 303c83a362da47ad97718d60bcc96b3a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 02:59:44.352487 8484 tablet_bootstrap.cc:654] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:44.353984 8485 tablet_bootstrap.cc:492] T 58268659f1e1460c821a93569fc0d6ab P ce0762f2fb2a41b4b7ae13351fdead66: No bootstrap required, opened a new log
I20260620 02:59:44.354163 8485 ts_tablet_manager.cc:1397] T 58268659f1e1460c821a93569fc0d6ab P ce0762f2fb2a41b4b7ae13351fdead66: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260620 02:59:44.355048 8484 tablet_bootstrap.cc:492] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0: No bootstrap required, opened a new log
I20260620 02:59:44.354988 8485 raft_consensus.cc:348] T 58268659f1e1460c821a93569fc0d6ab P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.355216 8484 ts_tablet_manager.cc:1397] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260620 02:59:44.355243 8485 raft_consensus.cc:374] T 58268659f1e1460c821a93569fc0d6ab P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:44.355326 8485 raft_consensus.cc:729] T 58268659f1e1460c821a93569fc0d6ab P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0762f2fb2a41b4b7ae13351fdead66, State: Initialized, Role: FOLLOWER
I20260620 02:59:44.355576 8485 consensus_queue.cc:260] T 58268659f1e1460c821a93569fc0d6ab P ce0762f2fb2a41b4b7ae13351fdead66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.355865 8484 raft_consensus.cc:348] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.356094 8484 raft_consensus.cc:374] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:44.356158 8485 ts_tablet_manager.cc:1428] T 58268659f1e1460c821a93569fc0d6ab P ce0762f2fb2a41b4b7ae13351fdead66: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:59:44.356165 8484 raft_consensus.cc:729] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a51aaa9abc45c99545598d6f5f04c0, State: Initialized, Role: FOLLOWER
I20260620 02:59:44.355990 8493 tablet_replica.cc:439] T 58268659f1e1460c821a93569fc0d6ab P ce0762f2fb2a41b4b7ae13351fdead66: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } } }
I20260620 02:59:44.356315 8493 tablet_replica.cc:442] T 58268659f1e1460c821a93569fc0d6ab P ce0762f2fb2a41b4b7ae13351fdead66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:59:44.356407 8484 consensus_queue.cc:260] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.356968 8484 ts_tablet_manager.cc:1428] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:59:44.356863 8491 tablet_replica.cc:439] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } } }
I20260620 02:59:44.357167 8491 tablet_replica.cc:442] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:59:44.468931 8491 raft_consensus.cc:482] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:59:44.469143 8491 raft_consensus.cc:504] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.469978 8491 leader_election.cc:288] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 303c83a362da47ad97718d60bcc96b3a (127.0.97.2:34783), ce0762f2fb2a41b4b7ae13351fdead66 (127.0.97.3:42707)
I20260620 02:59:44.470523 8270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58268659f1e1460c821a93569fc0d6ab" candidate_uuid: "16a51aaa9abc45c99545598d6f5f04c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303c83a362da47ad97718d60bcc96b3a" is_pre_election: true
I20260620 02:59:44.470558 8408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58268659f1e1460c821a93569fc0d6ab" candidate_uuid: "16a51aaa9abc45c99545598d6f5f04c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" is_pre_election: true
I20260620 02:59:44.470813 8270 raft_consensus.cc:2454] T 58268659f1e1460c821a93569fc0d6ab P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16a51aaa9abc45c99545598d6f5f04c0 in term 0.
I20260620 02:59:44.470829 8408 raft_consensus.cc:2454] T 58268659f1e1460c821a93569fc0d6ab P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16a51aaa9abc45c99545598d6f5f04c0 in term 0.
I20260620 02:59:44.471278 8064 leader_election.cc:302] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a51aaa9abc45c99545598d6f5f04c0, 303c83a362da47ad97718d60bcc96b3a; no voters:
I20260620 02:59:44.471576 8491 raft_consensus.cc:2781] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:59:44.471660 8491 raft_consensus.cc:482] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:59:44.471771 8491 raft_consensus.cc:3037] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:44.473701 8491 raft_consensus.cc:504] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.474236 8491 leader_election.cc:288] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 [CANDIDATE]: Term 1 election: Requested vote from peers 303c83a362da47ad97718d60bcc96b3a (127.0.97.2:34783), ce0762f2fb2a41b4b7ae13351fdead66 (127.0.97.3:42707)
I20260620 02:59:44.474648 8270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58268659f1e1460c821a93569fc0d6ab" candidate_uuid: "16a51aaa9abc45c99545598d6f5f04c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303c83a362da47ad97718d60bcc96b3a"
I20260620 02:59:44.474872 8270 raft_consensus.cc:3037] T 58268659f1e1460c821a93569fc0d6ab P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:44.474831 8408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58268659f1e1460c821a93569fc0d6ab" candidate_uuid: "16a51aaa9abc45c99545598d6f5f04c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce0762f2fb2a41b4b7ae13351fdead66"
I20260620 02:59:44.475028 8408 raft_consensus.cc:3037] T 58268659f1e1460c821a93569fc0d6ab P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:44.477751 8270 raft_consensus.cc:2454] T 58268659f1e1460c821a93569fc0d6ab P 303c83a362da47ad97718d60bcc96b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16a51aaa9abc45c99545598d6f5f04c0 in term 1.
I20260620 02:59:44.477751 8408 raft_consensus.cc:2454] T 58268659f1e1460c821a93569fc0d6ab P ce0762f2fb2a41b4b7ae13351fdead66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16a51aaa9abc45c99545598d6f5f04c0 in term 1.
I20260620 02:59:44.478135 8064 leader_election.cc:302] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a51aaa9abc45c99545598d6f5f04c0, 303c83a362da47ad97718d60bcc96b3a; no voters:
I20260620 02:59:44.478435 8491 raft_consensus.cc:2781] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:44.478694 8491 raft_consensus.cc:686] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 LEADER]: Becoming Leader. State: Replica: 16a51aaa9abc45c99545598d6f5f04c0, State: Running, Role: LEADER
I20260620 02:59:44.478930 8491 consensus_queue.cc:237] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.480444 8514 tablet_replica.cc:439] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0: TxnStatusTablet state changed. Reason: New leader 16a51aaa9abc45c99545598d6f5f04c0. Latest consensus state: current_term: 1 leader_uuid: "16a51aaa9abc45c99545598d6f5f04c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } } }
I20260620 02:59:44.480669 8514 tablet_replica.cc:442] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:59:44.481180 8527 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 02:59:44.481393 8527 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 02:59:44.481932 7972 catalog_manager.cc:5495] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16a51aaa9abc45c99545598d6f5f04c0 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "16a51aaa9abc45c99545598d6f5f04c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } health_report { overall_health: UNKNOWN } } }
I20260620 02:59:44.519497 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:44.512702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36653 (local address 127.0.0.1:57638)
0620 02:59:44.513001 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:44.513028 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:44.513181 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:44.513879 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:44.513883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:44.513893 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:59:44.514050 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:44.514057 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.515694 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.515698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:44.516648 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:44.516659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.517691 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.517695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:44.517841 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:44.517892 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:59:44.519018 (+ 1126us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:59:44.519035 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 02:59:44.519102 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 02:59:44.519220 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 02:59:44.519635 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:44.512722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57638 (local address 127.0.97.1:36653)
0620 02:59:44.513359 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:44.513371 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:44.513392 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:59:44.513492 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:44.513498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:44.513632 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:44.513800 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:59:44.514205 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.515497 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.516800 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.517496 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.517656 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:44.518087 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:59:44.518887 (+ 800us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:59:44.519051 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:44.519150 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 02:59:44.519240 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":349,"threads_started":1}
I20260620 02:59:44.523913 8270 raft_consensus.cc:1261] T 58268659f1e1460c821a93569fc0d6ab P 303c83a362da47ad97718d60bcc96b3a [term 1 FOLLOWER]: Refusing update from remote peer 16a51aaa9abc45c99545598d6f5f04c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:44.524245 8408 raft_consensus.cc:1261] T 58268659f1e1460c821a93569fc0d6ab P ce0762f2fb2a41b4b7ae13351fdead66 [term 1 FOLLOWER]: Refusing update from remote peer 16a51aaa9abc45c99545598d6f5f04c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:44.524641 8491 consensus_queue.cc:1035] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:44.525036 8514 consensus_queue.cc:1035] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:44.529170 8493 tablet_replica.cc:439] T 58268659f1e1460c821a93569fc0d6ab P ce0762f2fb2a41b4b7ae13351fdead66: TxnStatusTablet state changed. Reason: New leader 16a51aaa9abc45c99545598d6f5f04c0. Latest consensus state: current_term: 1 leader_uuid: "16a51aaa9abc45c99545598d6f5f04c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } } }
I20260620 02:59:44.529536 8493 tablet_replica.cc:442] T 58268659f1e1460c821a93569fc0d6ab P ce0762f2fb2a41b4b7ae13351fdead66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:59:44.533928 8489 tablet_replica.cc:439] T 58268659f1e1460c821a93569fc0d6ab P 303c83a362da47ad97718d60bcc96b3a: TxnStatusTablet state changed. Reason: New leader 16a51aaa9abc45c99545598d6f5f04c0. Latest consensus state: current_term: 1 leader_uuid: "16a51aaa9abc45c99545598d6f5f04c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } } }
I20260620 02:59:44.534222 8489 tablet_replica.cc:442] T 58268659f1e1460c821a93569fc0d6ab P 303c83a362da47ad97718d60bcc96b3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:59:44.534809 8531 mvcc.cc:205] Tried to move back new op lower bound from 7298762279002988544 to 7298762278827601920. Current Snapshot: MvccSnapshot[applied={T|T < 7298762279002988544}]
I20260620 02:59:44.535316 8491 tablet_replica.cc:439] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16a51aaa9abc45c99545598d6f5f04c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } } }
I20260620 02:59:44.535645 8491 tablet_replica.cc:442] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:59:44.540232 8514 tablet_replica.cc:439] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16a51aaa9abc45c99545598d6f5f04c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } } }
I20260620 02:59:44.540488 8514 tablet_replica.cc:442] T 58268659f1e1460c821a93569fc0d6ab P 16a51aaa9abc45c99545598d6f5f04c0: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:59:44.540993 8493 tablet_replica.cc:439] T 58268659f1e1460c821a93569fc0d6ab P ce0762f2fb2a41b4b7ae13351fdead66: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16a51aaa9abc45c99545598d6f5f04c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } } }
I20260620 02:59:44.541227 8493 tablet_replica.cc:442] T 58268659f1e1460c821a93569fc0d6ab P ce0762f2fb2a41b4b7ae13351fdead66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:59:44.547233 8489 tablet_replica.cc:439] T 58268659f1e1460c821a93569fc0d6ab P 303c83a362da47ad97718d60bcc96b3a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16a51aaa9abc45c99545598d6f5f04c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } } }
I20260620 02:59:44.547492 8489 tablet_replica.cc:442] T 58268659f1e1460c821a93569fc0d6ab P 303c83a362da47ad97718d60bcc96b3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:59:44.554749 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:44.547850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57642 (local address 127.0.97.1:36653)
0620 02:59:44.548021 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:44.548027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:44.549356 (+ 1329us) server_negotiation.cc:407] Connection header received
0620 02:59:44.549702 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:44.549708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:44.549787 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:44.549882 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:59:44.550417 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.551554 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.552870 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.553546 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.553708 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:44.554031 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:59:44.554149 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:59:44.554237 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:44.554336 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 02:59:44.554408 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 02:59:44.571648 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:44.564991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60571 (local address 127.0.97.1:36653)
0620 02:59:44.565193 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:44.565201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:44.565862 (+ 661us) server_negotiation.cc:407] Connection header received
0620 02:59:44.566046 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:44.566050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:44.566139 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:44.566224 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:59:44.566837 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.567562 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.569917 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.570395 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.570486 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:44.571020 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:59:44.571112 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:59:44.571180 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:44.571329 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 02:59:44.571383 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 02:59:44.572016 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:44.565017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36653 (local address 127.0.97.1:60571)
0620 02:59:44.565763 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:44.565782 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:44.565933 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:44.566266 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:44.566271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:44.566284 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:59:44.566684 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:44.566695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.567683 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.567689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:44.569749 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:44.569764 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.570579 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.570586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:44.570771 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:44.570850 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:59:44.571211 (+ 361us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:59:44.571222 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:59:44.571306 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 02:59:44.571458 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":373,"threads_started":1}
I20260620 02:59:44.581552 8270 raft_consensus.cc:1261] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a [term 1 FOLLOWER]: Refusing update from remote peer 16a51aaa9abc45c99545598d6f5f04c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:44.581619 8408 raft_consensus.cc:1261] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66 [term 1 FOLLOWER]: Refusing update from remote peer 16a51aaa9abc45c99545598d6f5f04c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:44.582158 8514 consensus_queue.cc:1035] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:44.582441 8491 consensus_queue.cc:1035] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:44.685324 7972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:44.685624 7972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:44.688611 7972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d: Sending DeleteTablet for 3 replicas of tablet 5efd979abbf84faba719beeb67a49c99
I20260620 02:59:44.689597 8111 tablet_service.cc:1514] Processing DeleteTablet for tablet 5efd979abbf84faba719beeb67a49c99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:44 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57628
I20260620 02:59:44.689605 8250 tablet_service.cc:1514] Processing DeleteTablet for tablet 5efd979abbf84faba719beeb67a49c99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:44 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52216
I20260620 02:59:44.689806 8389 tablet_service.cc:1514] Processing DeleteTablet for tablet 5efd979abbf84faba719beeb67a49c99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:44 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49824
I20260620 02:59:44.699555 8550 tablet_replica.cc:330] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a: stopping tablet replica
I20260620 02:59:44.699864 8550 raft_consensus.cc:2229] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:44.700207 8550 raft_consensus.cc:2258] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:44.703122 8550 ts_tablet_manager.cc:1905] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:44.705116 8552 tablet_replica.cc:330] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66: stopping tablet replica
I20260620 02:59:44.705317 8552 raft_consensus.cc:2229] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:44.705544 8552 raft_consensus.cc:2258] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:44.706073 8551 tablet_replica.cc:330] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0: stopping tablet replica
I20260620 02:59:44.706288 8551 raft_consensus.cc:2229] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:59:44.706684 8552 ts_tablet_manager.cc:1905] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:44.706907 8551 raft_consensus.cc:2258] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:44.707940 8551 ts_tablet_manager.cc:1905] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:44.713642 8552 ts_tablet_manager.cc:1918] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:59:44.713783 8552 log.cc:1196] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/5efd979abbf84faba719beeb67a49c99
I20260620 02:59:44.714208 8552 ts_tablet_manager.cc:1939] T 5efd979abbf84faba719beeb67a49c99 P ce0762f2fb2a41b4b7ae13351fdead66: Deleting consensus metadata
I20260620 02:59:44.715153 8551 ts_tablet_manager.cc:1918] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:59:44.715292 8551 log.cc:1196] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/5efd979abbf84faba719beeb67a49c99
I20260620 02:59:44.715389 7961 catalog_manager.cc:4841] TS ce0762f2fb2a41b4b7ae13351fdead66 (127.0.97.3:42707): tablet 5efd979abbf84faba719beeb67a49c99 (table test-table [id=b48d01351d2646e78272c6c71b176a2f]) successfully deleted
I20260620 02:59:44.715756 8551 ts_tablet_manager.cc:1939] T 5efd979abbf84faba719beeb67a49c99 P 16a51aaa9abc45c99545598d6f5f04c0: Deleting consensus metadata
I20260620 02:59:44.716814 7959 catalog_manager.cc:4841] TS 16a51aaa9abc45c99545598d6f5f04c0 (127.0.97.1:36653): tablet 5efd979abbf84faba719beeb67a49c99 (table test-table [id=b48d01351d2646e78272c6c71b176a2f]) successfully deleted
I20260620 02:59:44.720520 8550 ts_tablet_manager.cc:1918] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:59:44.720659 8550 log.cc:1196] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/5efd979abbf84faba719beeb67a49c99
I20260620 02:59:44.721112 8550 ts_tablet_manager.cc:1939] T 5efd979abbf84faba719beeb67a49c99 P 303c83a362da47ad97718d60bcc96b3a: Deleting consensus metadata
I20260620 02:59:44.722234 7961 catalog_manager.cc:4841] TS 303c83a362da47ad97718d60bcc96b3a (127.0.97.2:34783): tablet 5efd979abbf84faba719beeb67a49c99 (table test-table [id=b48d01351d2646e78272c6c71b176a2f]) successfully deleted
Jun 20 02:59:44 dist-test-slave-f1j0 krb5kdc[7929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:59:44 dist-test-slave-f1j0 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:44 dist-test-slave-f1j0 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:59:44.758741 8521 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:44.747461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40736 (local address 127.0.97.62:45219)
0620 02:59:44.747703 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:44.747710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:44.748091 (+ 381us) server_negotiation.cc:407] Connection header received
0620 02:59:44.748332 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:44.748337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:44.748400 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:44.748486 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:44.750148 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.750848 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.752566 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.752981 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.753068 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:44.758449 (+ 5381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40736: BlockingRecv error: recv got EOF from 127.0.0.1:40736 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 02:59:44 dist-test-slave-f1j0 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:59:44.776880 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:44.761907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40746 (local address 127.0.97.62:45219)
0620 02:59:44.762119 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:44.762125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:44.762438 (+ 313us) server_negotiation.cc:407] Connection header received
0620 02:59:44.762599 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:44.762602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:44.762657 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:44.762733 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:44.764174 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.764854 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.766670 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.767063 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.767150 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:44.771475 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:44.771481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:44.771484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:44.771515 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:44.774135 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:44.774809 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:44.774820 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:44.774823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:44.774963 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:44.775365 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:44.775373 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:44.775376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:44.775626 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:44.775898 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:44.776312 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 02:59:44.776534 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:59:44.780323 7972 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:59:44.780896 7972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:59:44.787907 8111 tablet_service.cc:1467] Processing CreateTablet for tablet b3361ec9d36247d88c183d5f5ede6475 (DEFAULT_TABLE table=test-table [id=4ee054440ed54bbd96bb885ac19a9c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:44.788411 8111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3361ec9d36247d88c183d5f5ede6475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:44.788416 8389 tablet_service.cc:1467] Processing CreateTablet for tablet b3361ec9d36247d88c183d5f5ede6475 (DEFAULT_TABLE table=test-table [id=4ee054440ed54bbd96bb885ac19a9c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:44.788470 8250 tablet_service.cc:1467] Processing CreateTablet for tablet b3361ec9d36247d88c183d5f5ede6475 (DEFAULT_TABLE table=test-table [id=4ee054440ed54bbd96bb885ac19a9c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:44.788802 8389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3361ec9d36247d88c183d5f5ede6475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:44.788900 8250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3361ec9d36247d88c183d5f5ede6475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:44.795091 8485 tablet_bootstrap.cc:492] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66: Bootstrap starting.
I20260620 02:59:44.795508 8484 tablet_bootstrap.cc:492] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0: Bootstrap starting.
I20260620 02:59:44.796789 8485 tablet_bootstrap.cc:654] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:44.796909 8484 tablet_bootstrap.cc:654] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:44.798980 8483 tablet_bootstrap.cc:492] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a: Bootstrap starting.
I20260620 02:59:44.799612 8485 tablet_bootstrap.cc:492] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66: No bootstrap required, opened a new log
I20260620 02:59:44.799770 8485 ts_tablet_manager.cc:1397] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260620 02:59:44.800501 8483 tablet_bootstrap.cc:654] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:44.800722 8485 raft_consensus.cc:348] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.801038 8485 raft_consensus.cc:374] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:44.801122 8485 raft_consensus.cc:729] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0762f2fb2a41b4b7ae13351fdead66, State: Initialized, Role: FOLLOWER
I20260620 02:59:44.801421 8485 consensus_queue.cc:260] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.802112 8485 ts_tablet_manager.cc:1428] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 02:59:44.809106 8483 tablet_bootstrap.cc:492] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a: No bootstrap required, opened a new log
I20260620 02:59:44.809288 8483 ts_tablet_manager.cc:1397] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 02:59:44.810127 8483 raft_consensus.cc:348] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.810423 8483 raft_consensus.cc:374] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:44.810520 8483 raft_consensus.cc:729] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303c83a362da47ad97718d60bcc96b3a, State: Initialized, Role: FOLLOWER
I20260620 02:59:44.810837 8484 tablet_bootstrap.cc:492] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0: No bootstrap required, opened a new log
I20260620 02:59:44.810775 8483 consensus_queue.cc:260] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.811009 8484 ts_tablet_manager.cc:1397] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.007s
I20260620 02:59:44.811396 8483 ts_tablet_manager.cc:1428] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 02:59:44.811842 8484 raft_consensus.cc:348] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.812172 8484 raft_consensus.cc:374] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:44.812281 8484 raft_consensus.cc:729] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a51aaa9abc45c99545598d6f5f04c0, State: Initialized, Role: FOLLOWER
I20260620 02:59:44.812548 8484 consensus_queue.cc:260] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.813036 8484 ts_tablet_manager.cc:1428] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 02:59:44.975961 8489 raft_consensus.cc:482] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:59:44.976191 8489 raft_consensus.cc:504] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.977672 8489 leader_election.cc:288] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16a51aaa9abc45c99545598d6f5f04c0 (127.0.97.1:36653), ce0762f2fb2a41b4b7ae13351fdead66 (127.0.97.3:42707)
I20260620 02:59:44.985307 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:44.978287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40593 (local address 127.0.97.3:42707)
0620 02:59:44.979460 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:44.979466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:44.979483 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:59:44.979545 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:44.979551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:44.979625 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:44.979725 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:44.980394 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.981375 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.983155 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.984302 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.984430 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:44.984540 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:44.984845 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 02:59:44.984991 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1041,"thread_start_us":924,"threads_started":1}
I20260620 02:59:44.985309 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:44.977924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42707 (local address 127.0.97.2:40593)
0620 02:59:44.978940 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:44.978955 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:44.979042 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:44.979773 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:44.979779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:44.979810 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:44.980231 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:44.980242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.981514 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.981521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:44.983006 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:44.983018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.984470 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.984477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:44.984664 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:44.984700 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:59:44.984803 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 02:59:44.984936 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921,"thread_start_us":770,"threads_started":1}
I20260620 02:59:44.986159 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:44.977885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36653 (local address 127.0.97.2:37813)
0620 02:59:44.978692 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:44.978712 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:44.978941 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:44.979480 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:44.979484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:44.979511 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:59:44.979762 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:44.979785 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.981160 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.981173 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:44.983279 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:44.983294 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.985288 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.985301 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:44.985501 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:44.985531 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 02:59:44.985705 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 02:59:44.985837 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":561,"threads_started":1}
I20260620 02:59:44.986174 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:44.978057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37813 (local address 127.0.97.1:36653)
0620 02:59:44.978310 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:44.978319 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:44.978828 (+ 509us) server_negotiation.cc:407] Connection header received
0620 02:59:44.979109 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:44.979116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:44.979206 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:44.979272 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:59:44.979989 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.981003 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.983442 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:44.984904 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:44.985026 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:44.985122 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:44.985732 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 02:59:44.985930 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:59:44.986302 8408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3361ec9d36247d88c183d5f5ede6475" candidate_uuid: "303c83a362da47ad97718d60bcc96b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" is_pre_election: true
I20260620 02:59:44.986548 8408 raft_consensus.cc:2454] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 303c83a362da47ad97718d60bcc96b3a in term 0.
I20260620 02:59:44.986771 8130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3361ec9d36247d88c183d5f5ede6475" candidate_uuid: "303c83a362da47ad97718d60bcc96b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a51aaa9abc45c99545598d6f5f04c0" is_pre_election: true
I20260620 02:59:44.987161 8130 raft_consensus.cc:2454] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 303c83a362da47ad97718d60bcc96b3a in term 0.
I20260620 02:59:44.987093 8204 leader_election.cc:302] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 303c83a362da47ad97718d60bcc96b3a, ce0762f2fb2a41b4b7ae13351fdead66; no voters:
I20260620 02:59:44.987545 8489 raft_consensus.cc:2781] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:59:44.987675 8489 raft_consensus.cc:482] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:59:44.987792 8489 raft_consensus.cc:3037] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:44.991019 8489 raft_consensus.cc:504] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:44.991751 8489 leader_election.cc:288] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [CANDIDATE]: Term 1 election: Requested vote from peers 16a51aaa9abc45c99545598d6f5f04c0 (127.0.97.1:36653), ce0762f2fb2a41b4b7ae13351fdead66 (127.0.97.3:42707)
I20260620 02:59:44.992195 8130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3361ec9d36247d88c183d5f5ede6475" candidate_uuid: "303c83a362da47ad97718d60bcc96b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a51aaa9abc45c99545598d6f5f04c0"
I20260620 02:59:44.992247 8408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3361ec9d36247d88c183d5f5ede6475" candidate_uuid: "303c83a362da47ad97718d60bcc96b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce0762f2fb2a41b4b7ae13351fdead66"
I20260620 02:59:44.992413 8130 raft_consensus.cc:3037] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:44.992446 8408 raft_consensus.cc:3037] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:44.995402 8408 raft_consensus.cc:2454] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 303c83a362da47ad97718d60bcc96b3a in term 1.
I20260620 02:59:44.995846 8204 leader_election.cc:302] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 303c83a362da47ad97718d60bcc96b3a, ce0762f2fb2a41b4b7ae13351fdead66; no voters:
I20260620 02:59:44.995639 8130 raft_consensus.cc:2454] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 303c83a362da47ad97718d60bcc96b3a in term 1.
I20260620 02:59:44.996127 8489 raft_consensus.cc:2781] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:44.997044 8489 raft_consensus.cc:686] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [term 1 LEADER]: Becoming Leader. State: Replica: 303c83a362da47ad97718d60bcc96b3a, State: Running, Role: LEADER
I20260620 02:59:44.997460 8489 consensus_queue.cc:237] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:45.001298 7973 catalog_manager.cc:5495] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 303c83a362da47ad97718d60bcc96b3a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "303c83a362da47ad97718d60bcc96b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } health_report { overall_health: UNKNOWN } } }
I20260620 02:59:45.053272 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:45.047642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.0.97.2:34783)
0620 02:59:45.048191 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:45.048196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:45.048213 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:59:45.048301 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:45.048304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:45.048378 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:45.048551 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:59:45.048968 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:45.049767 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:45.051081 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:45.051509 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:45.051640 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:45.052051 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:59:45.052655 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:59:45.052830 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:45.052876 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 02:59:45.052954 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":311,"threads_started":1}
I20260620 02:59:45.056699 8408 raft_consensus.cc:1261] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66 [term 1 FOLLOWER]: Refusing update from remote peer 303c83a362da47ad97718d60bcc96b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:45.056757 8130 raft_consensus.cc:1261] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 FOLLOWER]: Refusing update from remote peer 303c83a362da47ad97718d60bcc96b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:45.057475 8489 consensus_queue.cc:1035] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:45.057885 8571 consensus_queue.cc:1035] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:45.080960 7973 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:45.081215 7973 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:45.084393 7973 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d: Sending DeleteTablet for 3 replicas of tablet b3361ec9d36247d88c183d5f5ede6475
I20260620 02:59:45.085129 8250 tablet_service.cc:1514] Processing DeleteTablet for tablet b3361ec9d36247d88c183d5f5ede6475 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:45 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52216
I20260620 02:59:45.085140 8111 tablet_service.cc:1514] Processing DeleteTablet for tablet b3361ec9d36247d88c183d5f5ede6475 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:45 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57628
I20260620 02:59:45.085389 8550 tablet_replica.cc:330] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a: stopping tablet replica
I20260620 02:59:45.085400 8551 tablet_replica.cc:330] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0: stopping tablet replica
I20260620 02:59:45.085595 8550 raft_consensus.cc:2229] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:59:45.085618 8551 raft_consensus.cc:2229] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:45.085816 8551 raft_consensus.cc:2258] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:45.086104 8550 raft_consensus.cc:2258] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:45.086814 8389 tablet_service.cc:1514] Processing DeleteTablet for tablet b3361ec9d36247d88c183d5f5ede6475 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:45 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49824
I20260620 02:59:45.086956 8551 ts_tablet_manager.cc:1905] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:45.087061 8552 tablet_replica.cc:330] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66: stopping tablet replica
I20260620 02:59:45.087182 8550 ts_tablet_manager.cc:1905] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:45.087234 8552 raft_consensus.cc:2229] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:45.087433 8552 raft_consensus.cc:2258] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:45.088907 8552 ts_tablet_manager.cc:1905] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:45.093743 8550 ts_tablet_manager.cc:1918] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:59:45.093884 8550 log.cc:1196] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/b3361ec9d36247d88c183d5f5ede6475
I20260620 02:59:45.094301 8550 ts_tablet_manager.cc:1939] T b3361ec9d36247d88c183d5f5ede6475 P 303c83a362da47ad97718d60bcc96b3a: Deleting consensus metadata
I20260620 02:59:45.095335 8552 ts_tablet_manager.cc:1918] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:59:45.095317 7961 catalog_manager.cc:4841] TS 303c83a362da47ad97718d60bcc96b3a (127.0.97.2:34783): tablet b3361ec9d36247d88c183d5f5ede6475 (table test-table [id=4ee054440ed54bbd96bb885ac19a9c71]) successfully deleted
I20260620 02:59:45.095467 8552 log.cc:1196] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/b3361ec9d36247d88c183d5f5ede6475
I20260620 02:59:45.095853 8552 ts_tablet_manager.cc:1939] T b3361ec9d36247d88c183d5f5ede6475 P ce0762f2fb2a41b4b7ae13351fdead66: Deleting consensus metadata
I20260620 02:59:45.096758 7961 catalog_manager.cc:4841] TS ce0762f2fb2a41b4b7ae13351fdead66 (127.0.97.3:42707): tablet b3361ec9d36247d88c183d5f5ede6475 (table test-table [id=4ee054440ed54bbd96bb885ac19a9c71]) successfully deleted
I20260620 02:59:45.105819 8551 ts_tablet_manager.cc:1918] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:59:45.105996 8551 log.cc:1196] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/b3361ec9d36247d88c183d5f5ede6475
I20260620 02:59:45.106447 8551 ts_tablet_manager.cc:1939] T b3361ec9d36247d88c183d5f5ede6475 P 16a51aaa9abc45c99545598d6f5f04c0: Deleting consensus metadata
I20260620 02:59:45.107661 7959 catalog_manager.cc:4841] TS 16a51aaa9abc45c99545598d6f5f04c0 (127.0.97.1:36653): tablet b3361ec9d36247d88c183d5f5ede6475 (table test-table [id=4ee054440ed54bbd96bb885ac19a9c71]) successfully deleted
I20260620 02:59:45.107753 7973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:59:45.108453 7973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:59:45.117050 8111 tablet_service.cc:1467] Processing CreateTablet for tablet b885a2493fbf40ceab12a19b191fda53 (DEFAULT_TABLE table=test-table [id=c41319f2da6e443d96b057e8c0f24e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:45.117512 8111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b885a2493fbf40ceab12a19b191fda53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:45.120266 8250 tablet_service.cc:1467] Processing CreateTablet for tablet b885a2493fbf40ceab12a19b191fda53 (DEFAULT_TABLE table=test-table [id=c41319f2da6e443d96b057e8c0f24e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:45.120676 8250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b885a2493fbf40ceab12a19b191fda53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:45.125968 8389 tablet_service.cc:1467] Processing CreateTablet for tablet b885a2493fbf40ceab12a19b191fda53 (DEFAULT_TABLE table=test-table [id=c41319f2da6e443d96b057e8c0f24e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:59:45.126410 8389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b885a2493fbf40ceab12a19b191fda53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:45.136343 8485 tablet_bootstrap.cc:492] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66: Bootstrap starting.
I20260620 02:59:45.138003 8485 tablet_bootstrap.cc:654] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:45.139006 8483 tablet_bootstrap.cc:492] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a: Bootstrap starting.
I20260620 02:59:45.140738 8483 tablet_bootstrap.cc:654] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:45.141414 8484 tablet_bootstrap.cc:492] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0: Bootstrap starting.
I20260620 02:59:45.143082 8484 tablet_bootstrap.cc:654] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:45.143580 8485 tablet_bootstrap.cc:492] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66: No bootstrap required, opened a new log
I20260620 02:59:45.143756 8485 ts_tablet_manager.cc:1397] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260620 02:59:45.144630 8485 raft_consensus.cc:348] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:45.144979 8485 raft_consensus.cc:374] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:45.145072 8485 raft_consensus.cc:729] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0762f2fb2a41b4b7ae13351fdead66, State: Initialized, Role: FOLLOWER
I20260620 02:59:45.145416 8485 consensus_queue.cc:260] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:45.150034 8485 ts_tablet_manager.cc:1428] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260620 02:59:45.154796 8483 tablet_bootstrap.cc:492] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a: No bootstrap required, opened a new log
I20260620 02:59:45.155001 8483 ts_tablet_manager.cc:1397] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.005s
I20260620 02:59:45.156041 8484 tablet_bootstrap.cc:492] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0: No bootstrap required, opened a new log
I20260620 02:59:45.156229 8484 ts_tablet_manager.cc:1397] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260620 02:59:45.156931 8483 raft_consensus.cc:348] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:45.156993 8484 raft_consensus.cc:348] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:45.157233 8483 raft_consensus.cc:374] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:45.157248 8484 raft_consensus.cc:374] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:45.157321 8483 raft_consensus.cc:729] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303c83a362da47ad97718d60bcc96b3a, State: Initialized, Role: FOLLOWER
I20260620 02:59:45.157346 8484 raft_consensus.cc:729] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a51aaa9abc45c99545598d6f5f04c0, State: Initialized, Role: FOLLOWER
I20260620 02:59:45.157630 8484 consensus_queue.cc:260] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:45.157624 8483 consensus_queue.cc:260] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:45.160415 8484 ts_tablet_manager.cc:1428] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260620 02:59:45.159415 8483 ts_tablet_manager.cc:1428] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260620 02:59:45.342259 8571 raft_consensus.cc:482] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:59:45.342532 8571 raft_consensus.cc:504] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:45.343380 8571 leader_election.cc:288] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16a51aaa9abc45c99545598d6f5f04c0 (127.0.97.1:36653), ce0762f2fb2a41b4b7ae13351fdead66 (127.0.97.3:42707)
I20260620 02:59:45.344019 8130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b885a2493fbf40ceab12a19b191fda53" candidate_uuid: "303c83a362da47ad97718d60bcc96b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a51aaa9abc45c99545598d6f5f04c0" is_pre_election: true
I20260620 02:59:45.344103 8408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b885a2493fbf40ceab12a19b191fda53" candidate_uuid: "303c83a362da47ad97718d60bcc96b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" is_pre_election: true
I20260620 02:59:45.344311 8130 raft_consensus.cc:2454] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 303c83a362da47ad97718d60bcc96b3a in term 0.
I20260620 02:59:45.344367 8408 raft_consensus.cc:2454] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 303c83a362da47ad97718d60bcc96b3a in term 0.
I20260620 02:59:45.344877 8204 leader_election.cc:302] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 303c83a362da47ad97718d60bcc96b3a, ce0762f2fb2a41b4b7ae13351fdead66; no voters:
I20260620 02:59:45.345332 8571 raft_consensus.cc:2781] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:59:45.345444 8571 raft_consensus.cc:482] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:59:45.345556 8571 raft_consensus.cc:3037] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:45.349036 8571 raft_consensus.cc:504] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:45.350301 8408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b885a2493fbf40ceab12a19b191fda53" candidate_uuid: "303c83a362da47ad97718d60bcc96b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce0762f2fb2a41b4b7ae13351fdead66"
I20260620 02:59:45.350306 8130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b885a2493fbf40ceab12a19b191fda53" candidate_uuid: "303c83a362da47ad97718d60bcc96b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a51aaa9abc45c99545598d6f5f04c0"
I20260620 02:59:45.350513 8408 raft_consensus.cc:3037] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:45.350531 8130 raft_consensus.cc:3037] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:45.353488 8408 raft_consensus.cc:2454] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 303c83a362da47ad97718d60bcc96b3a in term 1.
I20260620 02:59:45.353897 8130 raft_consensus.cc:2454] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 303c83a362da47ad97718d60bcc96b3a in term 1.
I20260620 02:59:45.353983 8204 leader_election.cc:302] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 303c83a362da47ad97718d60bcc96b3a, ce0762f2fb2a41b4b7ae13351fdead66; no voters:
I20260620 02:59:45.354338 8489 raft_consensus.cc:2781] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:45.354370 8571 leader_election.cc:288] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [CANDIDATE]: Term 1 election: Requested vote from peers 16a51aaa9abc45c99545598d6f5f04c0 (127.0.97.1:36653), ce0762f2fb2a41b4b7ae13351fdead66 (127.0.97.3:42707)
I20260620 02:59:45.355060 8489 raft_consensus.cc:686] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [term 1 LEADER]: Becoming Leader. State: Replica: 303c83a362da47ad97718d60bcc96b3a, State: Running, Role: LEADER
I20260620 02:59:45.355388 8489 consensus_queue.cc:237] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } }
I20260620 02:59:45.359685 7972 catalog_manager.cc:5495] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 303c83a362da47ad97718d60bcc96b3a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "303c83a362da47ad97718d60bcc96b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "303c83a362da47ad97718d60bcc96b3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 } health_report { overall_health: UNKNOWN } } }
I20260620 02:59:45.421368 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:45.413933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46397 (local address 127.0.97.1:36653)
0620 02:59:45.414114 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:45.414123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:45.414389 (+ 266us) server_negotiation.cc:407] Connection header received
0620 02:59:45.414743 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:45.414753 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:45.414881 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:45.414999 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:59:45.415572 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:45.416726 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:45.419001 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:45.419730 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:45.419884 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:45.420476 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:59:45.420611 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:59:45.420710 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:45.420886 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 02:59:45.420976 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:59:45.421451 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:45.413515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36653 (local address 127.0.97.2:46397)
0620 02:59:45.414276 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:45.414303 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:45.414536 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:45.415038 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:45.415046 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:45.415060 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:59:45.415405 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:45.415415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:45.416883 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:45.416892 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:45.418844 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:45.418859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:45.419997 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:45.420004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:45.420193 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:45.420261 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:59:45.420750 (+ 489us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:59:45.420762 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 02:59:45.420840 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 02:59:45.420984 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":609,"threads_started":1}
I20260620 02:59:45.432849 8130 raft_consensus.cc:1261] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 FOLLOWER]: Refusing update from remote peer 303c83a362da47ad97718d60bcc96b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:45.433128 8408 raft_consensus.cc:1261] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66 [term 1 FOLLOWER]: Refusing update from remote peer 303c83a362da47ad97718d60bcc96b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:59:45.433719 8581 consensus_queue.cc:1035] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce0762f2fb2a41b4b7ae13351fdead66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:45.434125 8489 consensus_queue.cc:1035] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a51aaa9abc45c99545598d6f5f04c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:59:45.475471 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:45.466442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34783 (local address 127.0.97.1:40931)
0620 02:59:45.470035 (+ 3593us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:45.470053 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:45.470206 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:45.470529 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:45.470534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:45.470544 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:59:45.470886 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:45.470901 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:45.472352 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:45.472358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:45.473376 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:45.473386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:45.474205 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:45.474212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:45.474412 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:45.474468 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:59:45.474877 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:59:45.474889 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 02:59:45.474963 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 02:59:45.475104 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3439,"thread_start_us":3284,"threads_started":1}
I20260620 02:59:45.476267 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:45.466542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40931 (local address 127.0.97.2:34783)
0620 02:59:45.466697 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:45.466702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:45.470144 (+ 3442us) server_negotiation.cc:407] Connection header received
0620 02:59:45.470342 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:45.470348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:45.470423 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:45.470519 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:59:45.471058 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:45.472179 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:45.473517 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:45.474039 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:45.474174 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:45.474616 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:59:45.474780 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:59:45.474877 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:45.474983 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 02:59:45.475066 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 02:59:45.508642 7972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:45.508936 7972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:59:45.511981 7972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8bb4209e67c43ddb0af4b1500f7d16d: Sending DeleteTablet for 3 replicas of tablet b885a2493fbf40ceab12a19b191fda53
I20260620 02:59:45.512758 8111 tablet_service.cc:1514] Processing DeleteTablet for tablet b885a2493fbf40ceab12a19b191fda53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:45 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57628
I20260620 02:59:45.512854 8250 tablet_service.cc:1514] Processing DeleteTablet for tablet b885a2493fbf40ceab12a19b191fda53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:45 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52216
I20260620 02:59:45.512905 8389 tablet_service.cc:1514] Processing DeleteTablet for tablet b885a2493fbf40ceab12a19b191fda53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:59:45 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49824
I20260620 02:59:45.513063 8551 tablet_replica.cc:330] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0: stopping tablet replica
I20260620 02:59:45.513161 8552 tablet_replica.cc:330] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66: stopping tablet replica
I20260620 02:59:45.513242 8551 raft_consensus.cc:2229] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:45.513362 8552 raft_consensus.cc:2229] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:59:45.513453 8551 raft_consensus.cc:2258] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:45.513561 8552 raft_consensus.cc:2258] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:45.513733 8550 tablet_replica.cc:330] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a: stopping tablet replica
I20260620 02:59:45.513927 8550 raft_consensus.cc:2229] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:59:45.514266 8550 raft_consensus.cc:2258] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:59:45.515020 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 8045
I20260620 02:59:45.515301 8550 ts_tablet_manager.cc:1905] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:45.515379 8551 ts_tablet_manager.cc:1905] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:45.522996 8550 ts_tablet_manager.cc:1918] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:59:45.523164 8550 log.cc:1196] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/b885a2493fbf40ceab12a19b191fda53
I20260620 02:59:45.523455 8551 ts_tablet_manager.cc:1918] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:59:45.523607 8551 log.cc:1196] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/b885a2493fbf40ceab12a19b191fda53
I20260620 02:59:45.523638 8550 ts_tablet_manager.cc:1939] T b885a2493fbf40ceab12a19b191fda53 P 303c83a362da47ad97718d60bcc96b3a: Deleting consensus metadata
I20260620 02:59:45.524124 8551 ts_tablet_manager.cc:1939] T b885a2493fbf40ceab12a19b191fda53 P 16a51aaa9abc45c99545598d6f5f04c0: Deleting consensus metadata
I20260620 02:59:45.524912 7961 catalog_manager.cc:4841] TS 303c83a362da47ad97718d60bcc96b3a (127.0.97.2:34783): tablet b885a2493fbf40ceab12a19b191fda53 (table test-table [id=c41319f2da6e443d96b057e8c0f24e56]) successfully deleted
I20260620 02:59:45.525396 7959 catalog_manager.cc:4841] TS 16a51aaa9abc45c99545598d6f5f04c0 (127.0.97.1:36653): tablet b885a2493fbf40ceab12a19b191fda53 (table test-table [id=c41319f2da6e443d96b057e8c0f24e56]) successfully deleted
I20260620 02:59:45.526508 8552 ts_tablet_manager.cc:1905] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:59:45.533427 8552 ts_tablet_manager.cc:1918] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:59:45.533619 8552 log.cc:1196] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/b885a2493fbf40ceab12a19b191fda53
I20260620 02:59:45.534077 8552 ts_tablet_manager.cc:1939] T b885a2493fbf40ceab12a19b191fda53 P ce0762f2fb2a41b4b7ae13351fdead66: Deleting consensus metadata
I20260620 02:59:45.535272 7961 catalog_manager.cc:4841] TS ce0762f2fb2a41b4b7ae13351fdead66 (127.0.97.3:42707): tablet b885a2493fbf40ceab12a19b191fda53 (table test-table [id=c41319f2da6e443d96b057e8c0f24e56]) successfully deleted
Jun 20 02:59:45 dist-test-slave-f1j0 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260620 02:59:45.544008 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:45.520575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.0.97.1:36653)
0620 02:59:45.520799 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:45.520805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:45.526717 (+ 5912us) server_negotiation.cc:407] Connection header received
0620 02:59:45.526930 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:45.526937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:45.527033 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:45.527153 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:45.528576 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:45.529251 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:45.530544 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:45.530962 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:45.531098 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:45.538411 (+ 7313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:45.538425 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:45.538440 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:45.538478 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:45.541203 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:45.541936 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:45.541944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:45.541947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:45.542075 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:45.542477 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:45.542481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:45.542484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:45.542854 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:45.543157 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:45.543449 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 02:59:45.543689 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 02:59:45.544667 8176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:45.722359 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 8045
I20260620 02:59:45.749818 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 8185
Jun 20 02:59:45 dist-test-slave-f1j0 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260620 02:59:45.764192 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:45.750594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52224 (local address 127.0.97.2:34783)
0620 02:59:45.750755 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:45.750759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:45.750805 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:59:45.750985 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:45.750989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:45.751055 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:45.751143 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:45.752233 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:45.752903 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:45.754193 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:45.754579 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:45.754668 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:45.758728 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:45.758742 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:45.758756 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:45.758791 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:45.761559 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:45.762228 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:45.762238 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:45.762248 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:45.762364 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:45.762717 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:45.762723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:45.762727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:45.763062 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:45.763370 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:45.763653 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 02:59:45.763870 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 02:59:45.764796 8315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:45.887174 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 8185
I20260620 02:59:45.909662 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 8324
Jun 20 02:59:45 dist-test-slave-f1j0 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260620 02:59:45.924815 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:45.910474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49840 (local address 127.0.97.3:42707)
0620 02:59:45.910902 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:45.910907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:45.910923 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:59:45.910976 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:45.910980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:45.911048 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:45.911138 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:45.912416 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:45.913121 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:45.914587 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:45.915025 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:45.915131 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:45.919530 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:45.919545 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:45.919562 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:45.919597 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:45.922460 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:45.923047 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:45.923053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:45.923056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:45.923180 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:45.923485 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:45.923488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:45.923490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:45.923837 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:45.924059 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:45.924239 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 02:59:45.924417 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":227,"threads_started":1}
I20260620 02:59:45.925623 8454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:46.045105 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 8324
I20260620 02:59:46.066152 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 7942
I20260620 02:59:46.067250 8014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:46.184365 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 7942
2026-06-20T02:59:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6039 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:59:46 dist-test-slave-f1j0 krb5kdc[8601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:59:46 dist-test-slave-f1j0 krb5kdc[8601](info): set up 2 sockets
Jun 20 02:59:46 dist-test-slave-f1j0 krb5kdc[8601](info): commencing operation
krb5kdc: starting...
W20260620 02:59:47.274480 388 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 02:59:47 dist-test-slave-f1j0 krb5kdc[8601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:59:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:59:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:59:47.487277 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46785
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:47.812187 8614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:47.812477 8614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:47.812565 8614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:47.819577 8614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:59:47.819674 8614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:47.819718 8614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:47.819766 8614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:59:47.819813 8614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:59:47.829244 8614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:47.830446 8614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:47.831703 8614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:47.843304 8620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:47.843415 8619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:47.843364 8622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:47.844046 8614 server_base.cc:1034] running on GCE node
I20260620 02:59:47.844901 8614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:47.846410 8614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:47.847769 8614 hybrid_clock.cc:648] HybridClock initialized: now 1781924387847719 us; error 61 us; skew 500 ppm
I20260620 02:59:47.848137 8614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T02:59:47Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1730 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:59:48 dist-test-slave-f1j0 krb5kdc[8628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:59:48 dist-test-slave-f1j0 krb5kdc[8628](info): set up 2 sockets
Jun 20 02:59:48 dist-test-slave-f1j0 krb5kdc[8628](info): commencing operation
krb5kdc: starting...
W20260620 02:59:49.010610 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 02:59:49 dist-test-slave-f1j0 krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:59:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:59:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:59:49.226658 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39051
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:49.543154 8641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:49.543465 8641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:49.543628 8641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:49.551010 8641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:59:49.551110 8641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:49.551201 8641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:49.551280 8641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:59:49.551360 8641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:59:49.561210 8641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:49.562566 8641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:49.563863 8641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:49.575757 8646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:49.575798 8649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:49.575683 8647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:49.576783 8641 server_base.cc:1034] running on GCE node
I20260620 02:59:49.577520 8641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:49.578898 8641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:49.580127 8641 hybrid_clock.cc:648] HybridClock initialized: now 1781924389580087 us; error 49 us; skew 500 ppm
I20260620 02:59:49.580603 8641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:49 dist-test-slave-f1j0 krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:49.583788 8641 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:59:49.585353 8641 webserver.cc:460] Webserver started at http://127.0.97.62:35617/ using document root <none> and password file <none>
I20260620 02:59:49.586078 8641 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:49.586229 8641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:49.586541 8641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:49.589491 8641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "51009da04dea46afb732670b8bc5f598"
format_stamp: "Formatted at 2026-06-20 02:59:49 on dist-test-slave-f1j0"
server_key: "c24d08da3da59d94474790832dc1f0aa"
server_key_iv: "281390105977cde95bb1c32fe21259a1"
server_key_version: "encryptionkey@0"
I20260620 02:59:49.590348 8641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "51009da04dea46afb732670b8bc5f598"
format_stamp: "Formatted at 2026-06-20 02:59:49 on dist-test-slave-f1j0"
server_key: "c24d08da3da59d94474790832dc1f0aa"
server_key_iv: "281390105977cde95bb1c32fe21259a1"
server_key_version: "encryptionkey@0"
I20260620 02:59:49.595248 8641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:59:49.599381 8656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:49.600713 8641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:59:49.600919 8641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "51009da04dea46afb732670b8bc5f598"
format_stamp: "Formatted at 2026-06-20 02:59:49 on dist-test-slave-f1j0"
server_key: "c24d08da3da59d94474790832dc1f0aa"
server_key_iv: "281390105977cde95bb1c32fe21259a1"
server_key_version: "encryptionkey@0"
I20260620 02:59:49.601119 8641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:49.640309 8641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:49.644832 8641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:49.645146 8641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:49.662884 8641 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39051
I20260620 02:59:49.662907 8707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39051 every 8 connection(s)
I20260620 02:59:49.664275 8641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:59:49.668008 8708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:59:49.668355 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 8641
I20260620 02:59:49.668587 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 02:59:49.669392 388 external_mini_cluster.cc:1428] Setting key e86722f0178fb7be6d6dbaa907ebda80
I20260620 02:59:49.681798 8708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598: Bootstrap starting.
Jun 20 02:59:49 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:49.685348 8708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598: Neither blocks nor log segments found. Creating new log.
I20260620 02:59:49.686514 8708 log.cc:824] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:49.689527 8708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598: No bootstrap required, opened a new log
I20260620 02:59:49.692404 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:49.671436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58450 (local address 127.0.97.62:39051)
0620 02:59:49.672189 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:49.672201 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:49.672244 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:59:49.674217 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:49.674240 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:49.674636 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:49.674966 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:49.676688 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:49.678025 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:49.679789 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:49.680465 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:49.680593 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:49.685005 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:49.685013 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:49.685029 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:49.685066 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:49.688296 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:49.689034 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:49.689041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:49.689048 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:49.689209 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:49.689638 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:49.689643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:49.689647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:49.690210 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:49.690514 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:49.690951 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 02:59:49.691402 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":358,"threads_started":1}
I20260620 02:59:49.697856 8708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51009da04dea46afb732670b8bc5f598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39051 } }
I20260620 02:59:49.698210 8708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:59:49.698315 8708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51009da04dea46afb732670b8bc5f598, State: Initialized, Role: FOLLOWER
I20260620 02:59:49.698895 8708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51009da04dea46afb732670b8bc5f598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39051 } }
I20260620 02:59:49.699123 8708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:59:49.699260 8708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:59:49.699436 8708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:59:49.701820 8708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51009da04dea46afb732670b8bc5f598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39051 } }
I20260620 02:59:49.702301 8708 leader_election.cc:302] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51009da04dea46afb732670b8bc5f598; no voters:
I20260620 02:59:49.702869 8708 leader_election.cc:288] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:59:49.703006 8713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:59:49.703675 8713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [term 1 LEADER]: Becoming Leader. State: Replica: 51009da04dea46afb732670b8bc5f598, State: Running, Role: LEADER
I20260620 02:59:49.704217 8713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51009da04dea46afb732670b8bc5f598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39051 } }
I20260620 02:59:49.704985 8708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:59:49.707062 8714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51009da04dea46afb732670b8bc5f598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51009da04dea46afb732670b8bc5f598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39051 } } }
I20260620 02:59:49.707396 8714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:49.706794 8715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51009da04dea46afb732670b8bc5f598. Latest consensus state: current_term: 1 leader_uuid: "51009da04dea46afb732670b8bc5f598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51009da04dea46afb732670b8bc5f598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39051 } } }
I20260620 02:59:49.708398 8715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:49.709028 8718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:59:49.714210 8718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:59:49.723251 8718 catalog_manager.cc:1347] Generated new cluster ID: f61fb2bd9b7740399f7dfd0a699ef893
I20260620 02:59:49.723367 8718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:59:49.733460 8718 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:59:49.735296 8718 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:59:49.746599 8718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598: Generated new TSK 0
I20260620 02:59:49.747632 8718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:59:49.800750 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39051
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:50.146135 8734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:50.146438 8734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:50.146605 8734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:50.153986 8734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:50.154138 8734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:50.154330 8734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:59:50.163996 8734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:50.165293 8734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:50.166561 8734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:50.179327 8739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:50.179337 8742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:50.179349 8740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:50.180083 8734 server_base.cc:1034] running on GCE node
I20260620 02:59:50.180627 8734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:50.181459 8734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:50.182652 8734 hybrid_clock.cc:648] HybridClock initialized: now 1781924390182608 us; error 33 us; skew 500 ppm
I20260620 02:59:50.182971 8734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:50 dist-test-slave-f1j0 krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:50.186195 8734 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:59:50.189152 8734 webserver.cc:460] Webserver started at http://127.0.97.1:41437/ using document root <none> and password file <none>
I20260620 02:59:50.189898 8734 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:50.190037 8734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:50.190361 8734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:50.193467 8734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "27e520f42bfa4f5b995c182217cd271c"
format_stamp: "Formatted at 2026-06-20 02:59:50 on dist-test-slave-f1j0"
server_key: "32026399ff01839820540069177ee162"
server_key_iv: "c5e8ad530413fd54f9dc8ba23cccca70"
server_key_version: "encryptionkey@0"
I20260620 02:59:50.194305 8734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "27e520f42bfa4f5b995c182217cd271c"
format_stamp: "Formatted at 2026-06-20 02:59:50 on dist-test-slave-f1j0"
server_key: "32026399ff01839820540069177ee162"
server_key_iv: "c5e8ad530413fd54f9dc8ba23cccca70"
server_key_version: "encryptionkey@0"
I20260620 02:59:50.199337 8734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:59:50.202759 8749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:50.204073 8734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:59:50.204270 8734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "27e520f42bfa4f5b995c182217cd271c"
format_stamp: "Formatted at 2026-06-20 02:59:50 on dist-test-slave-f1j0"
server_key: "32026399ff01839820540069177ee162"
server_key_iv: "c5e8ad530413fd54f9dc8ba23cccca70"
server_key_version: "encryptionkey@0"
I20260620 02:59:50.204510 8734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:50.222971 8734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:50.226871 8734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:50.227144 8734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:50.228034 8734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:50.229306 8734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:50.229410 8734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:50.229553 8734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:50.229701 8734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 02:59:50.262375 8734 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38859
I20260620 02:59:50.262430 8861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38859 every 8 connection(s)
I20260620 02:59:50.263652 8734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:59:50.273864 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 8734
I20260620 02:59:50.274087 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:59:50.274837 388 external_mini_cluster.cc:1428] Setting key 182849b3d52ba9b20a7e2a433d54cb48
Jun 20 02:59:50 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:50.284386 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:50.266209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48051 (local address 127.0.97.62:39051)
0620 02:59:50.266649 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:50.266654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:50.267527 (+ 873us) server_negotiation.cc:407] Connection header received
0620 02:59:50.269279 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:50.269284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:50.269385 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:50.269504 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:50.270976 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:50.271676 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:50.273195 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:50.273659 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:50.273787 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:50.277605 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:50.277614 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:50.277622 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:50.277664 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:50.281250 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:50.282037 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:50.282045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:50.282052 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:50.282192 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:50.282660 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:50.282669 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:50.282675 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:50.283055 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:50.283201 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:50.283883 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 02:59:50.284128 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":231,"threads_started":1}
I20260620 02:59:50.285776 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:50.266684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.1:48051)
0620 02:59:50.267365 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:50.267406 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:50.269012 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:50.269752 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:50.269760 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:50.270283 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:50.270813 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:50.270829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:50.271847 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:50.271851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:50.273045 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:50.273056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:50.273879 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:50.273887 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:50.274168 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:50.274892 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:50.274917 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:50.277367 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:50.281417 (+ 4050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:50.281425 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:50.281436 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:50.281922 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:50.282333 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:50.282338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:50.282340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:50.282529 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:50.283193 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:50.283198 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:50.283506 (+ 308us) client_negotiation.cc:769] Sending connection context
0620 02:59:50.283736 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 02:59:50.284193 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":265,"threads_started":1}
I20260620 02:59:50.287233 8862 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39051
I20260620 02:59:50.287520 8862 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:50.288208 8862 heartbeater.cc:511] Master 127.0.97.62:39051 requested a full tablet report, sending...
I20260620 02:59:50.290197 8673 ts_manager.cc:194] Registered new tserver with Master: 27e520f42bfa4f5b995c182217cd271c (127.0.97.1:38859)
I20260620 02:59:50.291648 8673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48051
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:59:50.322871 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39051
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:50.642660 8868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:50.642977 8868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:50.643151 8868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:50.650484 8868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:50.650629 8868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:50.650818 8868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:59:50.660585 8868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:50.661892 8868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:50.663280 8868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:50.676252 8874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:50.676760 8873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:50.676811 8876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:50.677270 8868 server_base.cc:1034] running on GCE node
I20260620 02:59:50.677943 8868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:50.678892 8868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:50.680119 8868 hybrid_clock.cc:648] HybridClock initialized: now 1781924390680083 us; error 57 us; skew 500 ppm
I20260620 02:59:50.680460 8868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:50 dist-test-slave-f1j0 krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:50.686221 8868 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:59:50.687687 8868 webserver.cc:460] Webserver started at http://127.0.97.2:40221/ using document root <none> and password file <none>
I20260620 02:59:50.688409 8868 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:50.688560 8868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:50.688871 8868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:50.691877 8868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "17d0bcc6270240ccb928c40c7672ebe3"
format_stamp: "Formatted at 2026-06-20 02:59:50 on dist-test-slave-f1j0"
server_key: "260fd31ad7ac541af0a4e16c1334eb9b"
server_key_iv: "3505e90cec1bdc1cb1a8e3eac35da8c0"
server_key_version: "encryptionkey@0"
I20260620 02:59:50.692647 8868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "17d0bcc6270240ccb928c40c7672ebe3"
format_stamp: "Formatted at 2026-06-20 02:59:50 on dist-test-slave-f1j0"
server_key: "260fd31ad7ac541af0a4e16c1334eb9b"
server_key_iv: "3505e90cec1bdc1cb1a8e3eac35da8c0"
server_key_version: "encryptionkey@0"
I20260620 02:59:50.697733 8868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:59:50.700994 8883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:50.702337 8868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:59:50.702500 8868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "17d0bcc6270240ccb928c40c7672ebe3"
format_stamp: "Formatted at 2026-06-20 02:59:50 on dist-test-slave-f1j0"
server_key: "260fd31ad7ac541af0a4e16c1334eb9b"
server_key_iv: "3505e90cec1bdc1cb1a8e3eac35da8c0"
server_key_version: "encryptionkey@0"
I20260620 02:59:50.702697 8868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:50.728799 8868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:50.732744 8868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:50.733033 8868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:50.733991 8868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:50.735306 8868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:50.735409 8868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:50.735550 8868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:50.735626 8868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:50.768404 8868 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42849
I20260620 02:59:50.768450 8995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42849 every 8 connection(s)
I20260620 02:59:50.769775 8868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:59:50.773298 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 8868
I20260620 02:59:50.773492 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:59:50.774026 388 external_mini_cluster.cc:1428] Setting key 0c25f930fd867e30da8ecb46391ec1b1
Jun 20 02:59:50 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:50.790251 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:50.772293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43101 (local address 127.0.97.62:39051)
0620 02:59:50.772483 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:50.772488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:50.773836 (+ 1348us) server_negotiation.cc:407] Connection header received
0620 02:59:50.775652 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:50.775657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:50.775743 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:50.775852 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:50.777304 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:50.778048 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:50.779556 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:50.779977 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:50.780109 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:50.783510 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:50.783519 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:50.783527 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:50.783572 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:50.787070 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:50.787781 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:50.787789 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:50.787795 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:50.787943 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:50.788327 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:50.788334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:50.788341 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:50.788712 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:50.788874 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:50.789503 (+ 629us) server_negotiation.cc:299] Negotiation successful
0620 02:59:50.789793 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:59:50.790954 8998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:50.772798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.2:43101)
0620 02:59:50.773679 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:50.773717 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:50.775388 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:50.776085 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:50.776092 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:50.776586 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:50.777133 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:50.777150 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:50.778195 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:50.778201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:50.779406 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:50.779418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:50.780186 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:50.780193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:50.780418 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:50.781093 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:50.781115 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:50.783293 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:50.787213 (+ 3920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:50.787220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:50.787233 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:50.787653 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:50.788047 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:50.788050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:50.788053 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:50.788221 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:50.788846 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:50.788852 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:50.789153 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 02:59:50.789376 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 02:59:50.789837 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":315,"threads_started":1}
I20260620 02:59:50.792548 8996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39051
I20260620 02:59:50.792853 8996 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:50.793550 8996 heartbeater.cc:511] Master 127.0.97.62:39051 requested a full tablet report, sending...
I20260620 02:59:50.795099 8673 ts_manager.cc:194] Registered new tserver with Master: 17d0bcc6270240ccb928c40c7672ebe3 (127.0.97.2:42849)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:59:50.795938 8673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:59:50.824388 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39051
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:51.145991 9001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:51.146306 9001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:51.146472 9001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:51.153790 9001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:51.153918 9001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:51.154101 9001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:59:51.163792 9001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:51.165112 9001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:51.166497 9001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:51.179587 9006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:51.179651 9007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:51.179673 9009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:51.180223 9001 server_base.cc:1034] running on GCE node
I20260620 02:59:51.180795 9001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:51.181774 9001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:51.182973 9001 hybrid_clock.cc:648] HybridClock initialized: now 1781924391182921 us; error 62 us; skew 500 ppm
I20260620 02:59:51.183272 9001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:51 dist-test-slave-f1j0 krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:51.186897 9001 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:59:51.188369 9001 webserver.cc:460] Webserver started at http://127.0.97.3:45427/ using document root <none> and password file <none>
I20260620 02:59:51.189172 9001 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:51.189332 9001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:51.189742 9001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:59:51.193408 9001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "e8ca4a96b4b941fa9b99ec8e646ee082"
format_stamp: "Formatted at 2026-06-20 02:59:51 on dist-test-slave-f1j0"
server_key: "f177ef62b6ec0fc3e6eaa8caba92594c"
server_key_iv: "76e67785cdbbc54882915e6cf0175de5"
server_key_version: "encryptionkey@0"
I20260620 02:59:51.194239 9001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e8ca4a96b4b941fa9b99ec8e646ee082"
format_stamp: "Formatted at 2026-06-20 02:59:51 on dist-test-slave-f1j0"
server_key: "f177ef62b6ec0fc3e6eaa8caba92594c"
server_key_iv: "76e67785cdbbc54882915e6cf0175de5"
server_key_version: "encryptionkey@0"
I20260620 02:59:51.199033 9001 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:59:51.202376 9016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:51.203641 9001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:59:51.203816 9001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "e8ca4a96b4b941fa9b99ec8e646ee082"
format_stamp: "Formatted at 2026-06-20 02:59:51 on dist-test-slave-f1j0"
server_key: "f177ef62b6ec0fc3e6eaa8caba92594c"
server_key_iv: "76e67785cdbbc54882915e6cf0175de5"
server_key_version: "encryptionkey@0"
I20260620 02:59:51.204036 9001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:51.219710 9001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:51.223884 9001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:51.224165 9001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:51.225060 9001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:51.226347 9001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:51.226449 9001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:51.226603 9001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:51.226675 9001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:51.259429 9001 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38775
I20260620 02:59:51.259469 9128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38775 every 8 connection(s)
I20260620 02:59:51.260730 9001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:59:51.264497 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 9001
I20260620 02:59:51.264672 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:59:51.265175 388 external_mini_cluster.cc:1428] Setting key db5dc5489cc625e9ccc082e090b87366
Jun 20 02:59:51 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:59:51.281250 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:51.263351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36887 (local address 127.0.97.62:39051)
0620 02:59:51.263555 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:51.263562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:51.264871 (+ 1309us) server_negotiation.cc:407] Connection header received
0620 02:59:51.267391 (+ 2520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:51.267396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:51.267478 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:51.267582 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:51.269225 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:51.270047 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:51.271603 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:51.271961 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:51.272051 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:51.275363 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:51.275368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:51.275371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:51.275401 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:51.278013 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:51.278817 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:51.278823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:51.278828 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:51.278969 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:51.279472 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:51.279477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:51.279479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:51.279791 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:51.279951 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:51.280683 (+ 732us) server_negotiation.cc:299] Negotiation successful
0620 02:59:51.280893 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:59:51.282325 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:51.263851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.3:36887)
0620 02:59:51.264688 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:51.264739 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:51.267105 (+ 2366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:51.267857 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:51.267868 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:51.268523 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:51.269075 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:51.269092 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:51.270172 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:51.270176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:51.271474 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:51.271485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:51.272188 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:51.272193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:51.272384 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:51.272990 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:51.273011 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:51.275165 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:59:51.278129 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:51.278135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:51.278147 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:51.278679 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:51.279106 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:59:51.279110 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:59:51.279112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:59:51.279371 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:59:51.279936 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:59:51.279944 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:59:51.280284 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 02:59:51.280578 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 02:59:51.281091 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":363,"threads_started":1}
I20260620 02:59:51.283541 9129 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39051
I20260620 02:59:51.283841 9129 heartbeater.cc:464] Registering TS with master...
I20260620 02:59:51.284530 9129 heartbeater.cc:511] Master 127.0.97.62:39051 requested a full tablet report, sending...
I20260620 02:59:51.285872 8673 ts_manager.cc:194] Registered new tserver with Master: e8ca4a96b4b941fa9b99ec8e646ee082 (127.0.97.3:38775)
I20260620 02:59:51.286648 8673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36887
I20260620 02:59:51.292562 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:59:51.294552 8862 heartbeater.cc:503] Master 127.0.97.62:39051 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 02:59:51.425226 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 8734
Jun 20 02:59:51 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:59:51.441075 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:51.426441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47372 (local address 127.0.97.1:38859)
0620 02:59:51.427085 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:51.427092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:51.427119 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:59:51.427180 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:51.427186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:51.427400 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:51.427538 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:51.428787 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:51.429849 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:51.431013 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:51.431558 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:51.431653 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:51.435598 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:51.435609 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:51.435625 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:51.435661 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:51.438687 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:51.439300 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:51.439308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:51.439310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:51.439432 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:51.439836 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:51.439840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:51.439842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:51.440170 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:51.440391 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:51.440598 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 02:59:51.440792 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":344,"threads_started":1}
I20260620 02:59:51.442395 8857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:51.525245 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 8734
I20260620 02:59:51.541775 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 8868
Jun 20 02:59:51 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:59:51.557077 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:51.542700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43974 (local address 127.0.97.2:42849)
0620 02:59:51.543254 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:51.543268 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:51.543294 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:59:51.543355 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:51.543361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:51.543569 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:51.543695 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:51.544812 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:51.545880 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:51.547049 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:51.547575 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:51.547673 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:51.551683 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:51.551694 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:51.551708 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:51.551738 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:51.554373 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:51.555034 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:51.555042 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:51.555044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:51.555163 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:51.555518 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:51.555524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:51.555527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:51.555859 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:51.556162 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:51.556463 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 02:59:51.556721 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":296,"threads_started":1}
I20260620 02:59:51.558377 8991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:51.640626 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 8868
I20260620 02:59:51.656212 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 9001
Jun 20 02:59:51 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:59:51.673204 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:51.657248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.0.97.3:38775)
0620 02:59:51.657904 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:51.657911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:51.657938 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:59:51.658000 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:51.658006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:51.658210 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:51.658341 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:51.659562 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:51.660566 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:51.661920 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:51.662473 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:51.662643 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:51.667594 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:51.667606 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:51.667621 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:51.667653 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:51.670556 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:51.671219 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:51.671227 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:51.671229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:51.671359 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:51.671727 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:51.671733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:51.671736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:51.672061 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:51.672379 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:51.672616 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 02:59:51.672868 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":349,"threads_started":1}
I20260620 02:59:51.674530 9124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:51.756044 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 9001
I20260620 02:59:51.772032 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 8641
I20260620 02:59:51.773223 8703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:59:51.872920 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 8641
I20260620 02:59:51.893453 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39051
--webserver_interface=127.0.97.62
--webserver_port=35617
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:59:52.216054 9146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:52.216325 9146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:52.216511 9146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:52.223472 9146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:59:52.223567 9146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:52.223611 9146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:52.223699 9146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:59:52.223763 9146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:59:52.233214 9146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:52.234544 9146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:52.235883 9146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:52.247634 9154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:52.247800 9151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:52.247756 9152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:52.248934 9146 server_base.cc:1034] running on GCE node
I20260620 02:59:52.249826 9146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:52.251256 9146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:52.252482 9146 hybrid_clock.cc:648] HybridClock initialized: now 1781924392252439 us; error 35 us; skew 500 ppm
I20260620 02:59:52.252839 9146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:52 dist-test-slave-f1j0 krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:52.255817 9146 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:59:52.257300 9146 webserver.cc:460] Webserver started at http://127.0.97.62:35617/ using document root <none> and password file <none>
I20260620 02:59:52.258060 9146 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:52.258217 9146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:52.263584 9146 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 02:59:52.266294 9161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:52.267591 9146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:59:52.267779 9146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "51009da04dea46afb732670b8bc5f598"
format_stamp: "Formatted at 2026-06-20 02:59:49 on dist-test-slave-f1j0"
server_key: "c24d08da3da59d94474790832dc1f0aa"
server_key_iv: "281390105977cde95bb1c32fe21259a1"
server_key_version: "encryptionkey@0"
I20260620 02:59:52.268743 9146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:52.286823 9146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:52.290589 9146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:52.290863 9146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:52.308736 9146 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39051
I20260620 02:59:52.308754 9212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39051 every 8 connection(s)
I20260620 02:59:52.310225 9146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:59:52.314113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 9146
I20260620 02:59:52.315029 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:38859
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39051
--webserver_port=41437
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:59:52.318336 9213 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:59:52.323241 9213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598: Bootstrap starting.
I20260620 02:59:52.342166 9213 log.cc:824] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598: Log is configured to *not* fsync() on all Append() calls
I20260620 02:59:52.350925 9213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:59:52.351455 9213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598: Bootstrap complete.
I20260620 02:59:52.360430 9213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51009da04dea46afb732670b8bc5f598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39051 } }
I20260620 02:59:52.361338 9213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51009da04dea46afb732670b8bc5f598, State: Initialized, Role: FOLLOWER
I20260620 02:59:52.362025 9213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51009da04dea46afb732670b8bc5f598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39051 } }
I20260620 02:59:52.362260 9213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:59:52.362373 9213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:59:52.362572 9213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:59:52.366369 9213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51009da04dea46afb732670b8bc5f598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39051 } }
I20260620 02:59:52.366884 9213 leader_election.cc:302] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51009da04dea46afb732670b8bc5f598; no voters:
I20260620 02:59:52.367518 9213 leader_election.cc:288] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 02:59:52.367653 9217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:59:52.368738 9217 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [term 2 LEADER]: Becoming Leader. State: Replica: 51009da04dea46afb732670b8bc5f598, State: Running, Role: LEADER
I20260620 02:59:52.369379 9217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51009da04dea46afb732670b8bc5f598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39051 } }
I20260620 02:59:52.369814 9213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:59:52.371994 9219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51009da04dea46afb732670b8bc5f598. Latest consensus state: current_term: 2 leader_uuid: "51009da04dea46afb732670b8bc5f598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51009da04dea46afb732670b8bc5f598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39051 } } }
I20260620 02:59:52.372251 9218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "51009da04dea46afb732670b8bc5f598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51009da04dea46afb732670b8bc5f598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39051 } } }
I20260620 02:59:52.372613 9219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:52.372680 9218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598 [sys.catalog]: This master's current role is: LEADER
I20260620 02:59:52.374857 9223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:59:52.381062 9223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:59:52.382452 9223 catalog_manager.cc:1259] Loaded cluster ID: f61fb2bd9b7740399f7dfd0a699ef893
I20260620 02:59:52.382602 9223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:59:52.386595 9223 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:59:52.388047 9223 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 51009da04dea46afb732670b8bc5f598: Loaded TSK: 0
I20260620 02:59:52.389670 9223 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 02:59:52.655861 9215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:52.656131 9215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:52.656306 9215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:52.663976 9215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:52.664093 9215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:52.664268 9215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:59:52.674645 9215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:38859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41437
--tserver_master_addrs=127.0.97.62:39051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:52.675990 9215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:52.677346 9215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:52.692281 9239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:52.692323 9240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:52.692595 9242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:52.693463 9215 server_base.cc:1034] running on GCE node
I20260620 02:59:52.694144 9215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:52.695112 9215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:52.696337 9215 hybrid_clock.cc:648] HybridClock initialized: now 1781924392696284 us; error 45 us; skew 500 ppm
I20260620 02:59:52.696687 9215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:52 dist-test-slave-f1j0 krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924392, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:52.699977 9215 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 02:59:52.701437 9215 webserver.cc:460] Webserver started at http://127.0.97.1:41437/ using document root <none> and password file <none>
I20260620 02:59:52.702158 9215 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:52.702312 9215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:52.707676 9215 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:59:52.710503 9249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:52.711846 9215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:59:52.712013 9215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "27e520f42bfa4f5b995c182217cd271c"
format_stamp: "Formatted at 2026-06-20 02:59:50 on dist-test-slave-f1j0"
server_key: "32026399ff01839820540069177ee162"
server_key_iv: "c5e8ad530413fd54f9dc8ba23cccca70"
server_key_version: "encryptionkey@0"
I20260620 02:59:52.712936 9215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:52.738873 9215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:52.743080 9215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:52.743388 9215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:52.744400 9215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:52.745776 9215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:52.745896 9215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:52.746025 9215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:52.746111 9215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:52.779255 9215 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38859
I20260620 02:59:52.779346 9361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38859 every 8 connection(s)
I20260620 02:59:52.780740 9215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:59:52.789368 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 9215
I20260620 02:59:52.790102 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42849
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39051
--webserver_port=40221
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:59:52 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:52 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:52.799592 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:52.783708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.1:52971)
0620 02:59:52.784595 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:52.784639 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:52.786440 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:52.787700 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:52.787707 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:52.788231 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:52.788751 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:52.788767 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:52.791179 (+ 2412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:52.791182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:52.792383 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:52.792394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:52.793889 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:52.793897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:52.794183 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:52.794882 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:52.794903 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:52.798666 (+ 3763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":398,"threads_started":1}
W20260620 02:59:52.800331 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:52.783649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52971 (local address 127.0.97.62:39051)
0620 02:59:52.784342 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:52.784377 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:52.784756 (+ 379us) server_negotiation.cc:407] Connection header received
0620 02:59:52.786929 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:52.786942 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:52.787271 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:52.787583 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:52.788987 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:52.791023 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:52.792530 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:52.793658 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:52.793859 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:52.799136 (+ 5277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52971: BlockingRecv error: recv got EOF from 127.0.97.1:52971 (error 108)
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":263,"threads_started":1}
W20260620 02:59:52.800412 9362 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39051 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:39051: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 02:59:52 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:52 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:52.808955 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:52.800986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.1:53205)
0620 02:59:52.801154 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:52.801171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:52.801248 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:52.802089 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:52.802095 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:52.802454 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:52.802584 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:52.802592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:52.803672 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:52.803677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:52.804546 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:52.804556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:52.805220 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:52.805226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:52.805410 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:52.806003 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:52.806020 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:52.808645 (+ 2625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 02:59:52.809628 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:52.801127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53205 (local address 127.0.97.62:39051)
0620 02:59:52.801816 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:52.801822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:52.801843 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:59:52.801895 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:52.801899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:52.801951 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:52.802066 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:52.802755 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:52.803534 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:52.804683 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:52.805081 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:52.805211 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:52.808915 (+ 3704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53205: BlockingRecv error: recv got EOF from 127.0.97.1:53205 (error 108)
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":355,"threads_started":1}
Jun 20 02:59:52 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:52 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:52.817466 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:52.809650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.1:52121)
0620 02:59:52.809904 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:52.809923 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:52.810009 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:52.810383 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:52.810386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:52.810635 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:52.810762 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:52.810769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:52.812233 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:52.812236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:52.813092 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:52.813100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:52.814005 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:52.814009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:52.814121 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:52.814685 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:52.814701 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:52.817271 (+ 2570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139}
W20260620 02:59:52.817740 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:52.809857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52121 (local address 127.0.97.62:39051)
0620 02:59:52.810062 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:52.810070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:52.810090 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:59:52.810147 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:52.810155 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:52.810237 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:52.810357 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:52.810911 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:52.812103 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:52.813219 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:52.813886 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:52.814003 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:52.817490 (+ 3487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52121: BlockingRecv error: recv got EOF from 127.0.97.1:52121 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 02:59:52.817828 9362 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:59:53.119426 9366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:53.119704 9366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:53.119794 9366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:53.127151 9366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:53.127274 9366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:53.127389 9366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:59:53.137384 9366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40221
--tserver_master_addrs=127.0.97.62:39051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:53.138739 9366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:53.140028 9366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:53.152995 9372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:53.153033 9375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:53.153055 9373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:53.154239 9366 server_base.cc:1034] running on GCE node
I20260620 02:59:53.154861 9366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:53.155781 9366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:53.157025 9366 hybrid_clock.cc:648] HybridClock initialized: now 1781924393156983 us; error 71 us; skew 500 ppm
I20260620 02:59:53.157334 9366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:53 dist-test-slave-f1j0 krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924393, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:53.160513 9366 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 02:59:53.161916 9366 webserver.cc:460] Webserver started at http://127.0.97.2:40221/ using document root <none> and password file <none>
I20260620 02:59:53.162556 9366 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:53.162660 9366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:53.167778 9366 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 02:59:53.170461 9382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:53.171788 9366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:59:53.171998 9366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "17d0bcc6270240ccb928c40c7672ebe3"
format_stamp: "Formatted at 2026-06-20 02:59:50 on dist-test-slave-f1j0"
server_key: "260fd31ad7ac541af0a4e16c1334eb9b"
server_key_iv: "3505e90cec1bdc1cb1a8e3eac35da8c0"
server_key_version: "encryptionkey@0"
I20260620 02:59:53.172897 9366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:53.193713 9366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:53.198917 9366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:53.199209 9366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:53.200241 9366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:53.201625 9366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:53.201754 9366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:53.201867 9366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:53.201957 9366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:53.234761 9366 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42849
I20260620 02:59:53.234858 9494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42849 every 8 connection(s)
I20260620 02:59:53.236253 9366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:59:53.244107 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 9366
I20260620 02:59:53.244714 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38775
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39051
--webserver_port=45427
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:59:53 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:53 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:59:53.252597 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:53.238777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51141 (local address 127.0.97.62:39051)
0620 02:59:53.238958 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:53.238962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:53.240108 (+ 1146us) server_negotiation.cc:407] Connection header received
0620 02:59:53.241992 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:53.241997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:53.242099 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:53.242217 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:53.243804 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.244553 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.246309 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.246746 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.246887 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:53.252387 (+ 5500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51141: BlockingRecv error: recv got EOF from 127.0.97.2:51141 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:59:53.253770 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:53.239226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.2:51141)
0620 02:59:53.239933 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:53.239970 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:53.241708 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:53.242473 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:53.242481 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:53.243010 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:53.243589 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:53.243606 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.244719 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.244723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:53.246144 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:53.246156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.247000 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.247008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:53.247233 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:53.247991 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:53.248020 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:53.252188 (+ 4168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":279,"threads_started":1}
W20260620 02:59:53.254679 9495 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39051 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:39051: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 02:59:53 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:53 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:53.264885 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:53.255287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.2:52119)
0620 02:59:53.255487 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:53.255509 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:53.255624 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:53.255953 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:53.255961 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:53.256425 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:53.256634 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:53.256648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.257646 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.257654 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:53.259342 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:53.259360 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.260183 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.260193 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:53.260435 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:53.261061 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:53.261090 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:53.264670 (+ 3580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 02:59:53.265659 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:53.255449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52119 (local address 127.0.97.62:39051)
0620 02:59:53.255584 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:53.255588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:53.255604 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:59:53.255742 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:53.255746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:53.255816 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:53.255895 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:53.256772 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.257463 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.259523 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.260016 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.260142 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:53.264980 (+ 4838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52119: BlockingRecv error: recv got EOF from 127.0.97.2:52119 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 02:59:53 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:53 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:53.274729 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:53.265712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.2:48403)
0620 02:59:53.265971 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:53.265989 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:53.266109 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:53.266486 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:53.266490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:53.266789 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:53.266943 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:53.266951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.268517 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.268521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:53.269401 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:53.269410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.270404 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.270409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:53.270525 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:53.271140 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:53.271157 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:53.274447 (+ 3290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142}
W20260620 02:59:53.274775 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:53.265789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48403 (local address 127.0.97.62:39051)
0620 02:59:53.265991 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:53.265999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:53.266076 (+ 77us) server_negotiation.cc:407] Connection header received
0620 02:59:53.266236 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:53.266242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:53.266343 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:53.266462 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:53.267087 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.268359 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.269566 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.270271 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.270399 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:53.274618 (+ 4219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48403: BlockingRecv error: recv got EOF from 127.0.97.2:48403 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 02:59:53.275156 9495 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:59:53.582270 9498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:59:53.582537 9498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:59:53.582624 9498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:59:53.589949 9498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:59:53.590051 9498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:59:53.590167 9498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:59:53.599720 9498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45427
--tserver_master_addrs=127.0.97.62:39051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:59:53.600965 9498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:59:53.602231 9498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:59:53.615985 9503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:53.615978 9504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:59:53.616057 9506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:59:53.617727 9498 server_base.cc:1034] running on GCE node
I20260620 02:59:53.618340 9498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:59:53.619257 9498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:59:53.620489 9498 hybrid_clock.cc:648] HybridClock initialized: now 1781924393620443 us; error 65 us; skew 500 ppm
I20260620 02:59:53.620815 9498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:59:53 dist-test-slave-f1j0 krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924393, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:59:53.623978 9498 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 02:59:53.625380 9498 webserver.cc:460] Webserver started at http://127.0.97.3:45427/ using document root <none> and password file <none>
I20260620 02:59:53.626137 9498 fs_manager.cc:359] Metadata directory not provided
I20260620 02:59:53.626286 9498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:59:53.631625 9498 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 02:59:53.634311 9513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:53.635596 9498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:59:53.635768 9498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "e8ca4a96b4b941fa9b99ec8e646ee082"
format_stamp: "Formatted at 2026-06-20 02:59:51 on dist-test-slave-f1j0"
server_key: "f177ef62b6ec0fc3e6eaa8caba92594c"
server_key_iv: "76e67785cdbbc54882915e6cf0175de5"
server_key_version: "encryptionkey@0"
I20260620 02:59:53.636703 9498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:59:53.678195 9498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:59:53.682682 9498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:59:53.682987 9498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:59:53.683979 9498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:59:53.685253 9498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:59:53.685381 9498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:53.685513 9498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:59:53.685619 9498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:59:53.717813 9498 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38775
I20260620 02:59:53.717892 9625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38775 every 8 connection(s)
I20260620 02:59:53.719261 9498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:59:53.729705 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 9498
Jun 20 02:59:53 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:53 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:59:53.735417 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:53.721707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35491 (local address 127.0.97.62:39051)
0620 02:59:53.721891 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:53.721896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:53.723064 (+ 1168us) server_negotiation.cc:407] Connection header received
0620 02:59:53.724865 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:53.724870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:53.724956 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:53.725066 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:53.726623 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.727332 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.728895 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.729437 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.729587 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:53.735158 (+ 5571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35491: BlockingRecv error: recv got EOF from 127.0.97.3:35491 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:59:53.735953 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:53.722208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.3:35491)
0620 02:59:53.722912 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:53.722949 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:53.724575 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:53.725302 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:53.725311 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:53.725907 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:53.726450 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:53.726468 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.727488 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.727498 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:53.728740 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:53.728751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.729733 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.729739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:53.730007 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:53.730952 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:53.730980 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:53.734934 (+ 3954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":273,"threads_started":1}
W20260620 02:59:53.736630 9626 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39051 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:39051: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 02:59:53 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:53 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:53.745102 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:53.737350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.3:41285)
0620 02:59:53.737521 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:53.737539 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:53.737670 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:53.738008 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:53.738012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:53.738334 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:53.738456 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:53.738463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.739388 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.739398 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:53.740312 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:53.740322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.741120 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.741126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:53.741304 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:53.741848 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:53.741863 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:53.744847 (+ 2984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 02:59:53.745389 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:53.737420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41285 (local address 127.0.97.62:39051)
0620 02:59:53.737620 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:53.737626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:53.737643 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:59:53.737833 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:53.737837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:53.737888 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:53.737986 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:53.738581 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.739254 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.740437 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.740968 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.741138 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:53.745082 (+ 3944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41285: BlockingRecv error: recv got EOF from 127.0.97.3:41285 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:59:53.746758 9364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:53.730712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33844 (local address 127.0.97.62:39051)
0620 02:59:53.730929 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:53.730936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:53.731305 (+ 369us) server_negotiation.cc:407] Connection header received
0620 02:59:53.731490 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:53.731495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:53.731557 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:53.731657 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:53.733396 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.734548 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.736093 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.736740 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.736865 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:53.739552 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:59:53.739560 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:59:53.739577 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:59:53.739609 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:59:53.743106 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:53.743820 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:53.743829 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:53.743838 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:53.744054 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:59:53.744536 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:59:53.744541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:59:53.744544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:59:53.745335 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:59:53.745698 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0620 02:59:53.746112 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 02:59:53.746408 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 02:59:53 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:53 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:53.754235 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:53.746187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.3:40685)
0620 02:59:53.746753 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:53.746772 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:53.746880 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:53.747276 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:53.747280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:53.747612 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:53.747788 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:53.747798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.748964 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.748970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:53.750090 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:53.750111 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.750870 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.750876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:53.751041 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:53.751521 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:53.751537 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:53.754038 (+ 2501us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454}
W20260620 02:59:53.754392 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:53.746496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40685 (local address 127.0.97.62:39051)
0620 02:59:53.746775 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:53.746781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:53.746820 (+ 39us) server_negotiation.cc:407] Connection header received
0620 02:59:53.747040 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:53.747045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:53.747114 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:53.747214 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:53.747926 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.748829 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.750252 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.750724 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.750851 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:53.754201 (+ 3350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40685: BlockingRecv error: recv got EOF from 127.0.97.3:40685 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
W20260620 02:59:53.754565 9626 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 02:59:53 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:53 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:53.826915 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:53.818401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.1:45361)
0620 02:59:53.818893 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:53.818907 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:53.819063 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:53.819385 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:53.819389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:53.819854 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:53.820030 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:53.820038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.821041 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.821047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:53.822319 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:53.822330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.823108 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.823114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:53.823301 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:53.823844 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:53.823860 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:53.826698 (+ 2838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":272,"threads_started":1}
W20260620 02:59:53.827232 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:53.818427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45361 (local address 127.0.97.62:39051)
0620 02:59:53.818642 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:53.818650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:53.818999 (+ 349us) server_negotiation.cc:407] Connection header received
0620 02:59:53.819186 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:53.819191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:53.819254 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:53.819357 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:53.820173 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.820904 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.822452 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:53.822939 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:53.823077 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:53.826893 (+ 3816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45361: BlockingRecv error: recv got EOF from 127.0.97.1:45361 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 02:59:54 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:54 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:54.284358 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:54.275723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.2:43399)
0620 02:59:54.276235 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:54.276250 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:54.276413 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:54.276752 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:54.276757 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:54.277295 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:54.277505 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:54.277513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:54.278727 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:54.278734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:54.279917 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:54.279930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:54.280642 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:54.280649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:54.280823 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:54.281381 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:54.281398 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:54.284106 (+ 2708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":272,"threads_started":1}
W20260620 02:59:54.284533 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:54.275809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43399 (local address 127.0.97.62:39051)
0620 02:59:54.276039 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:54.276048 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:54.276342 (+ 294us) server_negotiation.cc:407] Connection header received
0620 02:59:54.276557 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:54.276561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:54.276619 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:54.276721 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:54.277687 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:54.278595 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:54.280055 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:54.280470 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:54.280607 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:54.284338 (+ 3731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43399: BlockingRecv error: recv got EOF from 127.0.97.2:43399 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 02:59:54 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:54 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:54.763724 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:54.755177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.3:55379)
0620 02:59:54.755705 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:54.755721 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:54.755848 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:54.756146 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:54.756149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:54.756576 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:54.756761 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:54.756768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:54.757946 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:54.757950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:54.758879 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:54.758889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:54.759753 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:54.759761 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:54.759914 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:54.760568 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:54.760585 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:54.763505 (+ 2920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":290,"threads_started":1}
W20260620 02:59:54.763963 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:54.755256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55379 (local address 127.0.97.62:39051)
0620 02:59:54.755416 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:54.755420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:54.755796 (+ 376us) server_negotiation.cc:407] Connection header received
0620 02:59:54.755946 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:54.755951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:54.756014 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:54.756102 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:54.756920 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:54.757804 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:54.759001 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:54.759560 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:54.759669 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:54.763685 (+ 4016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55379: BlockingRecv error: recv got EOF from 127.0.97.3:55379 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 02:59:54 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:54 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:54.836174 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:54.827828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.1:57843)
0620 02:59:54.828325 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:54.828341 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:54.828464 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:54.828790 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:54.828793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:54.829216 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:54.829362 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:54.829370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:54.830439 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:54.830442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:54.831351 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:54.831361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:54.832004 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:54.832008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:54.832115 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:54.832721 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:54.832742 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:54.835944 (+ 3202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":293,"threads_started":1}
W20260620 02:59:54.836360 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:54.827934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57843 (local address 127.0.97.62:39051)
0620 02:59:54.828216 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:54.828222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:54.828420 (+ 198us) server_negotiation.cc:407] Connection header received
0620 02:59:54.828565 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:54.828569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:54.828659 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:54.828742 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:54.829504 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:54.830329 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:54.831484 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:54.831872 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:54.831964 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:54.836136 (+ 4172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57843: BlockingRecv error: recv got EOF from 127.0.97.1:57843 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Jun 20 02:59:55 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:55 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:55.292855 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:55.285238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.2:56209)
0620 02:59:55.285756 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:55.285772 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:55.285878 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:55.286170 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:55.286173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:55.286552 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:55.286684 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:55.286691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:55.287649 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:55.287652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:55.288546 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:55.288557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:55.289208 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:55.289215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:55.289411 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:55.290047 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:55.290065 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:55.292625 (+ 2560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":287,"threads_started":1}
W20260620 02:59:55.293012 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:55.285320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56209 (local address 127.0.97.62:39051)
0620 02:59:55.285478 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:55.285483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:55.285839 (+ 356us) server_negotiation.cc:407] Connection header received
0620 02:59:55.285972 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:55.285975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:55.286038 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:55.286119 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:55.286817 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:55.287540 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:55.288658 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:55.289064 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:55.289152 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:55.292804 (+ 3652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56209: BlockingRecv error: recv got EOF from 127.0.97.2:56209 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 02:59:55 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:55 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:55.775244 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:55.764637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.3:36357)
0620 02:59:55.765178 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:55.765195 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:55.765302 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:55.765625 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:55.765628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:55.765995 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:55.766132 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:55.766140 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:55.767088 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:55.767092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:55.767983 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:55.767992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:55.768644 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:55.768648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:55.768753 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:55.771898 (+ 3145us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:55.771915 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:55.774913 (+ 2998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":321,"threads_started":1}
W20260620 02:59:55.775349 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:55.764794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36357 (local address 127.0.97.62:39051)
0620 02:59:55.764967 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:55.764971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:55.765291 (+ 320us) server_negotiation.cc:407] Connection header received
0620 02:59:55.765405 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:55.765408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:55.765469 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:55.765553 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:55.766245 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:55.766975 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:55.768103 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:55.768522 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:55.768614 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:55.775104 (+ 6490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36357: BlockingRecv error: recv got EOF from 127.0.97.3:36357 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 02:59:55 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:55 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:55.844607 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:55.837120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.1:39249)
0620 02:59:55.837654 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:55.837680 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:55.837786 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:55.838069 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:55.838072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:55.838397 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:55.838552 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:55.838559 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:55.839459 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:55.839462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:55.840337 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:55.840347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:55.841035 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:55.841039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:55.841157 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:55.841801 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:55.841820 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:55.844376 (+ 2556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":248,"threads_started":1}
W20260620 02:59:55.844784 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:55.837175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39249 (local address 127.0.97.62:39051)
0620 02:59:55.837387 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:55.837397 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:55.837754 (+ 357us) server_negotiation.cc:407] Connection header received
0620 02:59:55.837892 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:55.837895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:55.837954 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:55.838035 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:55.838665 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:55.839353 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:55.840497 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:55.840921 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:55.841011 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:55.844551 (+ 3540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39249: BlockingRecv error: recv got EOF from 127.0.97.1:39249 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 02:59:56 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:56 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:56.302160 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:56.293996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.2:41733)
0620 02:59:56.294506 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:56.294522 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:56.294678 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:56.295009 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:56.295013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:56.295363 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:56.295527 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:56.295534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:56.296529 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:56.296535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:56.297406 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:56.297417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:56.298124 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:56.298130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:56.298298 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:56.298986 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:56.299002 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:56.301812 (+ 2810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":275,"threads_started":1}
W20260620 02:59:56.302214 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:56.293991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41733 (local address 127.0.97.62:39051)
0620 02:59:56.294163 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:56.294167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:56.294612 (+ 445us) server_negotiation.cc:407] Connection header received
0620 02:59:56.294797 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:56.294802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:56.294882 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:56.294983 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:56.295666 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:56.296386 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:56.297540 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:56.297984 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:56.298114 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:56.302000 (+ 3886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41733: BlockingRecv error: recv got EOF from 127.0.97.2:41733 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 02:59:56 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:56 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:56.784953 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:56.776226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.3:45461)
0620 02:59:56.776750 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:56.776788 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:56.776949 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:56.777334 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:56.777341 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:56.777826 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:56.778028 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:56.778039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:56.779093 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:56.779099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:56.780005 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:56.780016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:56.780688 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:56.780693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:56.780867 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:56.781558 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:56.781597 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:56.784549 (+ 2952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":272,"threads_started":1}
W20260620 02:59:56.785022 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:56.776293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45461 (local address 127.0.97.62:39051)
0620 02:59:56.776504 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:56.776513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:56.776876 (+ 363us) server_negotiation.cc:407] Connection header received
0620 02:59:56.777080 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:56.777086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:56.777196 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:56.777292 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:56.778156 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:56.778951 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:56.780147 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:56.780546 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:56.780678 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:56.784771 (+ 4093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45461: BlockingRecv error: recv got EOF from 127.0.97.3:45461 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 02:59:56 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:56 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:56.853703 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:56.845575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.1:49453)
0620 02:59:56.846163 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:56.846183 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:56.846294 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:56.846738 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:56.846742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:56.847171 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:56.847366 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:56.847379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:56.848419 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:56.848422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:56.849360 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:56.849370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:56.849996 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:56.850000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:56.850118 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:56.850691 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:56.850708 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:56.853435 (+ 2727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":326,"threads_started":1}
W20260620 02:59:56.853878 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:56.845693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49453 (local address 127.0.97.62:39051)
0620 02:59:56.845903 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:56.845907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:56.846428 (+ 521us) server_negotiation.cc:407] Connection header received
0620 02:59:56.846486 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:56.846490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:56.846592 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:56.846694 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:56.847519 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:56.848303 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:56.849476 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:56.849880 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:56.849963 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:56.853679 (+ 3716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49453: BlockingRecv error: recv got EOF from 127.0.97.1:49453 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 02:59:57 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:57 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:57.311069 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:57.303079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.2:57751)
0620 02:59:57.303520 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:57.303535 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:57.303640 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:57.303915 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:57.303918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:57.304239 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:57.304401 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:57.304409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:57.305315 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:57.305318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:57.306345 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:57.306357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:57.307016 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:57.307020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:57.307140 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:57.307689 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:57.307705 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:57.310770 (+ 3065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":244,"threads_started":1}
W20260620 02:59:57.311214 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:57.303126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57751 (local address 127.0.97.62:39051)
0620 02:59:57.303379 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:57.303384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:57.303603 (+ 219us) server_negotiation.cc:407] Connection header received
0620 02:59:57.303743 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:57.303746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:57.303808 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:57.303881 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:57.304507 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:57.305200 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:57.306462 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:57.306898 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:57.306996 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:57.310976 (+ 3980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57751: BlockingRecv error: recv got EOF from 127.0.97.2:57751 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 02:59:57 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:57 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:57.793964 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:57.786043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.3:60601)
0620 02:59:57.786551 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:57.786567 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:57.786680 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:57.786986 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:57.786989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:57.787331 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:57.787481 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:57.787489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:57.788453 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:57.788456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:57.789426 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:57.789438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:57.790088 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:57.790092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:57.790221 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:57.790817 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:57.790841 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:57.793613 (+ 2772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":276,"threads_started":1}
W20260620 02:59:57.794030 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:57.786125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60601 (local address 127.0.97.62:39051)
0620 02:59:57.786302 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:57.786307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:57.786639 (+ 332us) server_negotiation.cc:407] Connection header received
0620 02:59:57.786777 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:57.786781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:57.786858 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:57.786944 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:57.787616 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:57.788346 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:57.789547 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:57.789968 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:57.790051 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:57.793799 (+ 3748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60601: BlockingRecv error: recv got EOF from 127.0.97.3:60601 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 02:59:57 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:57 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:57.862644 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:57.854632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.1:40927)
0620 02:59:57.855178 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:57.855195 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:57.855310 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:57.855573 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:57.855576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:57.855935 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:57.856086 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:57.856094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:57.857093 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:57.857096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:57.858066 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:57.858077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:57.858744 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:57.858749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:57.858903 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:57.859496 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:57.859514 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:57.862355 (+ 2841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":286,"threads_started":1}
W20260620 02:59:57.862850 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:57.854703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40927 (local address 127.0.97.62:39051)
0620 02:59:57.854875 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:57.854879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:57.855278 (+ 399us) server_negotiation.cc:407] Connection header received
0620 02:59:57.855395 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:57.855398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:57.855456 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:57.855534 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:57.856199 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:57.856968 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:57.858183 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:57.858581 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:57.858671 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:57.862566 (+ 3895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40927: BlockingRecv error: recv got EOF from 127.0.97.1:40927 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 02:59:58 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:58 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:58.320633 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:58.312070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.2:58959)
0620 02:59:58.312625 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:58.312639 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:58.312790 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:58.313096 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:58.313100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:58.313536 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:58.313737 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:58.313745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:58.314824 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:58.314830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:58.315718 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:58.315728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:58.316498 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:58.316504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:58.316676 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:58.317297 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:58.317317 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:58.320411 (+ 3094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":309,"threads_started":1}
W20260620 02:59:58.320767 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:58.312214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58959 (local address 127.0.97.62:39051)
0620 02:59:58.312411 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:58.312418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:58.312721 (+ 303us) server_negotiation.cc:407] Connection header received
0620 02:59:58.312921 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:58.312925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:58.312978 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:58.313069 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:58.313878 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:58.314651 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:58.315859 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:58.316351 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:58.316472 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:58.320580 (+ 4108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58959: BlockingRecv error: recv got EOF from 127.0.97.2:58959 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 02:59:58 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:58 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:58.802881 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:58.794892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.3:35287)
0620 02:59:58.795412 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:58.795428 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:58.795536 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:58.795844 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:58.795851 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:58.796351 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:58.796539 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:58.796551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:58.797806 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:58.797810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:58.798704 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:58.798725 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:58.799408 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:58.799412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:58.799523 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:58.800043 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:58.800060 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:58.802636 (+ 2576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":264,"threads_started":1}
W20260620 02:59:58.803005 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:58.795105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35287 (local address 127.0.97.62:39051)
0620 02:59:58.795280 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:58.795285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:58.795500 (+ 215us) server_negotiation.cc:407] Connection header received
0620 02:59:58.795630 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:58.795633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:58.795689 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:58.795773 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:58.796686 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:58.797639 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:58.798834 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:58.799282 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:58.799381 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:58.802807 (+ 3426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35287: BlockingRecv error: recv got EOF from 127.0.97.3:35287 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 02:59:58 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:58 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:58.871666 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:58.863572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.1:37269)
0620 02:59:58.864057 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:58.864072 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:58.864211 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:58.864523 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:58.864527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:58.864916 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:58.865060 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:58.865069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:58.866128 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:58.866136 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:58.867070 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:58.867089 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:58.867772 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:58.867778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:58.867949 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:58.868556 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:58.868573 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:58.871341 (+ 2768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":290,"threads_started":1}
W20260620 02:59:58.871796 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:58.863643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37269 (local address 127.0.97.62:39051)
0620 02:59:58.863837 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:58.863845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:58.864146 (+ 301us) server_negotiation.cc:407] Connection header received
0620 02:59:58.864344 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:58.864347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:58.864404 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:58.864496 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:58.865219 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:58.865983 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:58.867212 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:58.867627 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:58.867756 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:58.871544 (+ 3788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37269: BlockingRecv error: recv got EOF from 127.0.97.1:37269 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 02:59:59 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:59 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:59.328900 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:59.321464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.2:57801)
0620 02:59:59.321968 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:59.321984 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:59.322093 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:59.322387 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:59.322389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:59.322736 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:59.322887 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:59.322895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:59.323871 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:59.323875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:59.324762 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:59.324771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:59.325397 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:59.325401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:59.325519 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:59.326076 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:59.326093 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:59.328677 (+ 2584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":292,"threads_started":1}
W20260620 02:59:59.329068 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:59.321721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57801 (local address 127.0.97.62:39051)
0620 02:59:59.321929 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:59.321936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:59.322056 (+ 120us) server_negotiation.cc:407] Connection header received
0620 02:59:59.322188 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:59.322192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:59.322268 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:59.322345 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:59.323001 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:59.323757 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:59.324878 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:59.325279 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:59.325365 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:59.328860 (+ 3495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57801: BlockingRecv error: recv got EOF from 127.0.97.2:57801 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 02:59:59 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:59 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:59.811858 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:59.803802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.3:55531)
0620 02:59:59.804326 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:59.804343 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:59.804462 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:59.804764 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:59.804767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:59.805113 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:59.805266 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:59.805275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:59.806243 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:59.806247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:59.807158 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:59.807170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:59.807812 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:59.807815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:59.807929 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:59.808584 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:59.808602 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:59.811641 (+ 3039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":307,"threads_started":1}
W20260620 02:59:59.812049 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:59.803888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55531 (local address 127.0.97.62:39051)
0620 02:59:59.804045 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:59.804050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:59.804411 (+ 361us) server_negotiation.cc:407] Connection header received
0620 02:59:59.804563 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:59.804567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:59.804624 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:59.804711 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:59.805381 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:59.806132 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:59.807277 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:59.807692 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:59.807777 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:59.811851 (+ 4074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55531: BlockingRecv error: recv got EOF from 127.0.97.3:55531 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 02:59:59 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:59:59 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:59:59.880461 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:59:59.872628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.1:36203)
0620 02:59:59.873191 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 02:59:59.873207 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:59:59.873326 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:59:59.873643 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:59:59.873647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:59:59.874024 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:59:59.874180 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:59.874188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:59.875099 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:59.875102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:59.875992 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:59:59.876002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:59:59.876756 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:59:59.876760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:59:59.876869 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:59.877393 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:59:59.877409 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:59:59.880123 (+ 2714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":218,"threads_started":1}
W20260620 02:59:59.880617 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:59:59.872667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36203 (local address 127.0.97.62:39051)
0620 02:59:59.872852 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 02:59:59.872861 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:59:59.873285 (+ 424us) server_negotiation.cc:407] Connection header received
0620 02:59:59.873418 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:59:59.873421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:59:59.873490 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:59:59.873605 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:59:59.874297 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:59.874988 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:59.876108 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:59:59.876615 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:59:59.876700 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:59:59.880370 (+ 3670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36203: BlockingRecv error: recv got EOF from 127.0.97.1:36203 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 03:00:00 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:00 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:00.337956 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:00.329903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.2:58931)
0620 03:00:00.330536 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:00.330553 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:00.330669 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:00.330944 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:00.330947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:00.331340 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:00.331506 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:00.331514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:00.332439 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:00.332443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:00.333373 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:00.333384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:00.334092 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:00.334096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:00.334208 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:00.334791 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:00.334808 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:00.337548 (+ 2740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":314,"threads_started":1}
W20260620 03:00:00.338002 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:00.329960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58931 (local address 127.0.97.62:39051)
0620 03:00:00.330154 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:00.330158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:00.330622 (+ 464us) server_negotiation.cc:407] Connection header received
0620 03:00:00.330762 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:00.330765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:00.330826 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:00.330905 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:00.331618 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:00.332295 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:00.333487 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:00.333963 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:00.334053 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:00.337791 (+ 3738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58931: BlockingRecv error: recv got EOF from 127.0.97.2:58931 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 20 03:00:00 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:00 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:00.823120 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:00.812744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.3:42939)
0620 03:00:00.813332 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:00.813348 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:00.813461 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:00.813767 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:00.813771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:00.814177 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:00.814347 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:00.814356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:00.815314 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:00.815319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:00.816263 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:00.816275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:00.816989 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:00.816993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:00.817105 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:00.820044 (+ 2939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:00.820062 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:00.822886 (+ 2824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":256,"threads_started":1}
W20260620 03:00:00.823355 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:00.812850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42939 (local address 127.0.97.62:39051)
0620 03:00:00.813030 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:00.813035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:00.813421 (+ 386us) server_negotiation.cc:407] Connection header received
0620 03:00:00.813553 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:00.813556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:00.813635 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:00.813713 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:00.814461 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:00.815170 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:00.816417 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:00.816861 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:00.816952 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:00.823084 (+ 6132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42939: BlockingRecv error: recv got EOF from 127.0.97.3:42939 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 03:00:00 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:00 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:00.890115 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:00.881483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.1:49411)
0620 03:00:00.882015 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:00.882036 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:00.882233 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:00.882571 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:00.882576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:00.883034 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:00.883183 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:00.883191 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:00.884285 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:00.884292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:00.885283 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:00.885313 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:00.886350 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:00.886355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:00.886548 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:00.887129 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:00.887146 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:00.889887 (+ 2741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":293,"threads_started":1}
W20260620 03:00:00.890467 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:00.881802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49411 (local address 127.0.97.62:39051)
0620 03:00:00.882008 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:00.882014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:00.882129 (+ 115us) server_negotiation.cc:407] Connection header received
0620 03:00:00.882365 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:00.882371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:00.882437 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:00.882541 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:00.883357 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:00.884140 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:00.885447 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:00.886140 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:00.886309 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:00.890100 (+ 3791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49411: BlockingRecv error: recv got EOF from 127.0.97.1:49411 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 03:00:01 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:01 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:01.347029 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:01.338900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.2:33333)
0620 03:00:01.339373 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:01.339389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:01.339535 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:01.339847 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:01.339851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:01.340287 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:01.340426 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:01.340434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:01.341588 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:01.341598 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:01.342499 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:01.342510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:01.343229 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:01.343237 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:01.343406 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:01.343985 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:01.344002 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:01.346809 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":248,"threads_started":1}
W20260620 03:00:01.347280 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:01.338984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33333 (local address 127.0.97.62:39051)
0620 03:00:01.339267 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:01.339273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:01.339465 (+ 192us) server_negotiation.cc:407] Connection header received
0620 03:00:01.339653 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:01.339659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:01.339723 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:01.339820 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:01.340568 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:01.341431 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:01.342639 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:01.343053 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:01.343198 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:01.346988 (+ 3790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33333: BlockingRecv error: recv got EOF from 127.0.97.2:33333 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 03:00:01 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:01 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:01.831753 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:01.824092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.3:38025)
0620 03:00:01.824611 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:01.824628 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:01.824736 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:01.825014 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:01.825018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:01.825388 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:01.825531 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:01.825539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:01.826535 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:01.826538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:01.827434 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:01.827444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:01.828091 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:01.828096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:01.828222 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:01.828840 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:01.828855 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:01.831528 (+ 2673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":316,"threads_started":1}
W20260620 03:00:01.831957 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:01.824199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38025 (local address 127.0.97.62:39051)
0620 03:00:01.824380 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:01.824385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:01.824698 (+ 313us) server_negotiation.cc:407] Connection header received
0620 03:00:01.824841 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:01.824844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:01.824902 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:01.824974 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:01.825697 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:01.826415 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:01.827550 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:01.827944 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:01.828031 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:01.831715 (+ 3684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38025: BlockingRecv error: recv got EOF from 127.0.97.3:38025 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 03:00:01 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:01 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:01.899379 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:01.891134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.1:53915)
0620 03:00:01.891614 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:01.891630 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:01.891754 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:01.892049 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:01.892052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:01.892416 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:01.892574 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:01.892582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:01.893630 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:01.893646 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:01.894699 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:01.894713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:01.895434 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:01.895440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:01.895625 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:01.896238 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:01.896255 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:01.899134 (+ 2879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":269,"threads_started":1}
W20260620 03:00:01.899603 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:01.891207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53915 (local address 127.0.97.62:39051)
0620 03:00:01.891441 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:01.891447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:01.891715 (+ 268us) server_negotiation.cc:407] Connection header received
0620 03:00:01.891865 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:01.891869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:01.891931 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:01.892006 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:01.892691 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:01.893434 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:01.894843 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:01.895264 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:01.895425 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:01.899345 (+ 3920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53915: BlockingRecv error: recv got EOF from 127.0.97.1:53915 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 03:00:02 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:02 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:02.355914 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:02.347941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.2:50111)
0620 03:00:02.348417 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:02.348434 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:02.348549 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:02.348829 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:02.348832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:02.349186 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:02.349348 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:02.349355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:02.350327 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:02.350330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:02.351389 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:02.351400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:02.352101 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:02.352107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:02.352268 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:02.352806 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:02.352823 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:02.355554 (+ 2731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":267,"threads_started":1}
W20260620 03:00:02.355954 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:02.347977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50111 (local address 127.0.97.62:39051)
0620 03:00:02.348220 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:02.348229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:02.348517 (+ 288us) server_negotiation.cc:407] Connection header received
0620 03:00:02.348647 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:02.348651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:02.348712 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:02.348803 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:02.349461 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:02.350218 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:02.351528 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:02.351959 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:02.352086 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:02.355743 (+ 3657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50111: BlockingRecv error: recv got EOF from 127.0.97.2:50111 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 03:00:02 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:02 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:02.840953 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:02.832797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.3:34429)
0620 03:00:02.833277 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:02.833296 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:02.833445 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:02.833775 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:02.833779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:02.834177 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:02.834339 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:02.834347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:02.835335 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:02.835342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:02.836264 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:02.836274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:02.837037 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:02.837043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:02.837209 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:02.837914 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:02.837937 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:02.840566 (+ 2629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":247,"threads_started":1}
W20260620 03:00:02.841008 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:02.832864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34429 (local address 127.0.97.62:39051)
0620 03:00:02.833085 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:02.833093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:02.833379 (+ 286us) server_negotiation.cc:407] Connection header received
0620 03:00:02.833587 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:02.833592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:02.833650 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:02.833747 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:02.834476 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:02.835203 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:02.836399 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:02.836882 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:02.837023 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:02.840781 (+ 3758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34429: BlockingRecv error: recv got EOF from 127.0.97.3:34429 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 03:00:02 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:02 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:02.908794 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:02.900318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.1:45099)
0620 03:00:02.900843 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:02.900859 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:02.900965 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:02.901264 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:02.901268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:02.901664 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:02.901803 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:02.901810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:02.903110 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:02.903121 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:02.904384 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:02.904404 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:02.905055 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:02.905060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:02.905195 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:02.905873 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:02.905891 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:02.908581 (+ 2690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":295,"threads_started":1}
W20260620 03:00:02.908987 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:02.900550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45099 (local address 127.0.97.62:39051)
0620 03:00:02.900742 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:02.900746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:02.900932 (+ 186us) server_negotiation.cc:407] Connection header received
0620 03:00:02.901060 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:02.901064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:02.901125 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:02.901211 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:02.901926 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:02.902955 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:02.904510 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:02.904911 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:02.905005 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:02.908784 (+ 3779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45099: BlockingRecv error: recv got EOF from 127.0.97.1:45099 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 03:00:03 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:03 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:03.366231 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:03.357154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.2:46887)
0620 03:00:03.357770 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:03.357786 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:03.357945 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:03.358356 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:03.358361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:03.358829 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:03.359014 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:03.359026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:03.360351 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:03.360359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:03.361469 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:03.361481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:03.362242 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:03.362249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:03.362367 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:03.363026 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:03.363043 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:03.366014 (+ 2971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":320,"threads_started":1}
W20260620 03:00:03.366501 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:03.357237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46887 (local address 127.0.97.62:39051)
0620 03:00:03.357506 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:03.357514 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:03.357877 (+ 363us) server_negotiation.cc:407] Connection header received
0620 03:00:03.358100 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:03.358104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:03.358166 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:03.358318 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:03.359218 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:03.360165 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:03.361635 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:03.362069 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:03.362208 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:03.366196 (+ 3988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46887: BlockingRecv error: recv got EOF from 127.0.97.2:46887 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 03:00:03 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:03 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:03.849900 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:03.842009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.3:58381)
0620 03:00:03.842565 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:03.842581 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:03.842686 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:03.842969 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:03.842972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:03.843317 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:03.843458 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:03.843467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:03.844391 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:03.844395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:03.845340 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:03.845350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:03.846065 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:03.846069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:03.846184 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:03.846794 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:03.846811 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:03.849646 (+ 2835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":323,"threads_started":1}
W20260620 03:00:03.850061 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:03.842140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58381 (local address 127.0.97.62:39051)
0620 03:00:03.842328 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:03.842332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:03.842649 (+ 317us) server_negotiation.cc:407] Connection header received
0620 03:00:03.842777 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:03.842781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:03.842847 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:03.842927 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:03.843588 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:03.844287 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:03.845453 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:03.845927 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:03.846023 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:03.849844 (+ 3821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58381: BlockingRecv error: recv got EOF from 127.0.97.3:58381 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 03:00:03 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:03 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:03.918470 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:03.909859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.1:37867)
0620 03:00:03.910343 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:03.910358 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:03.910507 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:03.910839 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:03.910844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:03.911294 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:03.911466 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:03.911474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:03.912488 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:03.912494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:03.913514 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:03.913526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:03.914357 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:03.914363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:03.914528 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:03.915174 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:03.915199 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:03.918109 (+ 2910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":252,"threads_started":1}
W20260620 03:00:03.918641 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:03.909918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37867 (local address 127.0.97.62:39051)
0620 03:00:03.910129 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:03.910137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:03.910437 (+ 300us) server_negotiation.cc:407] Connection header received
0620 03:00:03.910632 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:03.910637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:03.910701 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:03.910815 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:03.911604 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:03.912359 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:03.913680 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:03.914184 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:03.914318 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:03.918329 (+ 4011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37867: BlockingRecv error: recv got EOF from 127.0.97.1:37867 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 03:00:04 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:04 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:04.375437 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:04.367180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.2:47983)
0620 03:00:04.367675 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:04.367691 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:04.367860 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:04.368193 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:04.368198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:04.368762 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:04.368928 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:04.368936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:04.370052 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:04.370059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:04.370949 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:04.370959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:04.371651 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:04.371656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:04.371801 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:04.372354 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:04.372371 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:04.375113 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":268,"threads_started":1}
W20260620 03:00:04.375551 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:04.367257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47983 (local address 127.0.97.62:39051)
0620 03:00:04.367485 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:04.367493 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:04.367788 (+ 295us) server_negotiation.cc:407] Connection header received
0620 03:00:04.367993 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:04.367996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:04.368054 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:04.368164 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:04.369065 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:04.369929 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:04.371083 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:04.371487 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:04.371631 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:04.375352 (+ 3721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47983: BlockingRecv error: recv got EOF from 127.0.97.2:47983 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 03:00:04 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:04 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:04.858855 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:04.850791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.3:40535)
0620 03:00:04.851350 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:04.851366 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:04.851475 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:04.851775 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:04.851779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:04.852178 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:04.852370 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:04.852379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:04.853379 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:04.853383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:04.854400 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:04.854413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:04.855040 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:04.855045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:04.855165 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:04.855777 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:04.855794 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:04.858594 (+ 2800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":337,"threads_started":1}
W20260620 03:00:04.859025 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:04.850878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40535 (local address 127.0.97.62:39051)
0620 03:00:04.851064 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:04.851068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:04.851435 (+ 367us) server_negotiation.cc:407] Connection header received
0620 03:00:04.851563 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:04.851567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:04.851623 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:04.851702 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:04.852489 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:04.853233 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:04.854523 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:04.854906 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:04.854990 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:04.858787 (+ 3797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40535: BlockingRecv error: recv got EOF from 127.0.97.3:40535 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 03:00:04 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:04 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:04.927802 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:04.919354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.1:57271)
0620 03:00:04.920041 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:04.920062 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:04.920217 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:04.920675 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:04.920678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:04.921212 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:04.921366 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:04.921375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:04.922430 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:04.922435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:04.923386 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:04.923397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:04.924072 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:04.924076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:04.924248 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:04.924798 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:04.924815 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:04.927575 (+ 2760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":446,"threads_started":1}
W20260620 03:00:04.927945 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:04.919462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57271 (local address 127.0.97.62:39051)
0620 03:00:04.919652 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:04.919660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:04.920148 (+ 488us) server_negotiation.cc:407] Connection header received
0620 03:00:04.920353 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:04.920357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:04.920416 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:04.920493 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:04.921487 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:04.922291 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:04.923509 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:04.923925 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:04.924023 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:04.927756 (+ 3733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57271: BlockingRecv error: recv got EOF from 127.0.97.1:57271 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 03:00:05 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:05 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:05.385000 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:05.376427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.2:60285)
0620 03:00:05.376938 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:05.376953 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:05.377101 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:05.377423 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:05.377427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:05.377857 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:05.378004 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:05.378013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:05.378999 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:05.379006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:05.379909 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:05.379920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:05.380681 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:05.380687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:05.380963 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:05.381537 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:05.381555 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:05.384679 (+ 3124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":272,"threads_started":1}
W20260620 03:00:05.385140 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:05.376481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60285 (local address 127.0.97.62:39051)
0620 03:00:05.376700 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:05.376709 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:05.377042 (+ 333us) server_negotiation.cc:407] Connection header received
0620 03:00:05.377231 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:05.377236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:05.377297 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:05.377393 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:05.378143 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:05.378865 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:05.380045 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:05.380531 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:05.380659 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:05.384897 (+ 4238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60285: BlockingRecv error: recv got EOF from 127.0.97.2:60285 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 03:00:05 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:05 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:05.870730 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:05.859784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.3:34823)
0620 03:00:05.860296 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:05.860312 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:05.860421 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:05.860714 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:05.860718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:05.861072 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:05.861249 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:05.861258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:05.862173 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:05.862177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:05.863077 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:05.863088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:05.863728 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:05.863732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:05.863858 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:05.867339 (+ 3481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:05.867355 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:05.870508 (+ 3153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":280,"threads_started":1}
W20260620 03:00:05.871102 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:05.859874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34823 (local address 127.0.97.62:39051)
0620 03:00:05.860036 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:05.860041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:05.860388 (+ 347us) server_negotiation.cc:407] Connection header received
0620 03:00:05.860515 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:05.860518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:05.860579 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:05.860661 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:05.861361 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:05.862063 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:05.863219 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:05.863609 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:05.863701 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:05.870677 (+ 6976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34823: BlockingRecv error: recv got EOF from 127.0.97.3:34823 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 03:00:05 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:05 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:05.936697 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:05.928779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.1:45289)
0620 03:00:05.929295 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:05.929310 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:05.929456 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:05.929788 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:05.929793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:05.930176 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:05.930334 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:05.930342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:05.931330 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:05.931338 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:05.932245 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:05.932265 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:05.933006 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:05.933013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:05.933176 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:05.933808 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:05.933826 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:05.936457 (+ 2631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":265,"threads_started":1}
W20260620 03:00:05.936914 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:05.928827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45289 (local address 127.0.97.62:39051)
0620 03:00:05.929063 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:05.929070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:05.929393 (+ 323us) server_negotiation.cc:407] Connection header received
0620 03:00:05.929585 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:05.929593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:05.929663 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:05.929767 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:05.930470 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:05.931192 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:05.932390 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:05.932831 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:05.932960 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:05.936649 (+ 3689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45289: BlockingRecv error: recv got EOF from 127.0.97.1:45289 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 03:00:06 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:06 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:06.394373 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:06.385943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.2:37011)
0620 03:00:06.386460 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:06.386475 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:06.386585 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:06.386923 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:06.386927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:06.387343 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:06.387513 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:06.387522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:06.388532 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:06.388536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:06.389460 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:06.389471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:06.390351 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:06.390356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:06.390472 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:06.391050 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:06.391067 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:06.394026 (+ 2959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":268,"threads_started":1}
W20260620 03:00:06.394483 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:06.386134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37011 (local address 127.0.97.62:39051)
0620 03:00:06.386348 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:06.386352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:06.386545 (+ 193us) server_negotiation.cc:407] Connection header received
0620 03:00:06.386670 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:06.386673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:06.386735 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:06.386820 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:06.387664 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:06.388397 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:06.389661 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:06.390153 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:06.390287 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:06.394237 (+ 3950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37011: BlockingRecv error: recv got EOF from 127.0.97.2:37011 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 20 03:00:06 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:06 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:06.879914 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:06.871678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.3:39853)
0620 03:00:06.872232 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:06.872249 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:06.872358 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:06.872629 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:06.872632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:06.873082 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:06.873230 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:06.873238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:06.874205 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:06.874209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:06.875134 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:06.875144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:06.875828 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:06.875833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:06.875940 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:06.876498 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:06.876522 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:06.879581 (+ 3059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":262,"threads_started":1}
W20260620 03:00:06.880033 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:06.871687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39853 (local address 127.0.97.62:39051)
0620 03:00:06.871888 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:06.871898 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:06.872307 (+ 409us) server_negotiation.cc:407] Connection header received
0620 03:00:06.872455 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:06.872458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:06.872518 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:06.872597 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:06.873353 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:06.874097 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:06.875246 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:06.875641 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:06.875785 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:06.879784 (+ 3999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39853: BlockingRecv error: recv got EOF from 127.0.97.3:39853 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 03:00:06 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:06 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:06.945267 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:06.937626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.1:33317)
0620 03:00:06.938112 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:06.938132 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:06.938265 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:06.938519 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:06.938522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:06.938866 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:06.939032 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:06.939039 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:06.939979 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:06.939983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:06.940974 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:06.940985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:06.941702 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:06.941706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:06.941826 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:06.942394 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:06.942410 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:06.945012 (+ 2602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":255,"threads_started":1}
W20260620 03:00:06.945426 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:06.937683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33317 (local address 127.0.97.62:39051)
0620 03:00:06.938119 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:06.938125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:06.938195 (+ 70us) server_negotiation.cc:407] Connection header received
0620 03:00:06.938367 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:06.938370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:06.938430 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:06.938506 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:06.939141 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:06.939850 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:06.941105 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:06.941533 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:06.941688 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:06.945206 (+ 3518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33317: BlockingRecv error: recv got EOF from 127.0.97.1:33317 (error 108)
Metrics: {"server-negotiator.queue_time_us":304}
Jun 20 03:00:07 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:07 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:07.403929 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:07.395374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.2:60785)
0620 03:00:07.395950 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:07.395965 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:07.396127 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:07.396477 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:07.396481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:07.396897 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:07.397070 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:07.397078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:07.398183 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:07.398190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:07.399099 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:07.399110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:07.399836 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:07.399842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:07.400006 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:07.400651 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:07.400668 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:07.403527 (+ 2859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":325,"threads_started":1}
W20260620 03:00:07.404007 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:07.395551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60785 (local address 127.0.97.62:39051)
0620 03:00:07.395814 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:07.395821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:07.396052 (+ 231us) server_negotiation.cc:407] Connection header received
0620 03:00:07.396270 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:07.396275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:07.396342 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:07.396460 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:07.397212 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:07.398044 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:07.399253 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:07.399655 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:07.399809 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:07.403760 (+ 3951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60785: BlockingRecv error: recv got EOF from 127.0.97.2:60785 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 03:00:07 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:07 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:07.889266 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:07.880807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.3:43233)
0620 03:00:07.881285 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:07.881300 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:07.881456 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:07.881811 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:07.881816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:07.882194 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:07.882355 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:07.882363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:07.883435 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:07.883441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:07.884404 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:07.884415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:07.885093 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:07.885099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:07.885269 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:07.885960 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:07.885976 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:07.889032 (+ 3056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":243,"threads_started":1}
W20260620 03:00:07.889428 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:07.880878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43233 (local address 127.0.97.62:39051)
0620 03:00:07.881105 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:07.881113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:07.881386 (+ 273us) server_negotiation.cc:407] Connection header received
0620 03:00:07.881610 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:07.881614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:07.881671 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:07.881773 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:07.882489 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:07.883308 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:07.884539 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:07.884950 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:07.885080 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:07.889214 (+ 4134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43233: BlockingRecv error: recv got EOF from 127.0.97.3:43233 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 03:00:07 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:07 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:07.954185 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:07.946413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.1:38035)
0620 03:00:07.946981 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:07.946996 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:07.947103 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:07.947415 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:07.947419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:07.947796 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:07.947974 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:07.947982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:07.948942 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:07.948945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:07.949856 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:07.949867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:07.950469 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:07.950473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:07.950579 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:07.951120 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:07.951137 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:07.953911 (+ 2774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":374,"threads_started":1}
W20260620 03:00:07.954299 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:07.946434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38035 (local address 127.0.97.62:39051)
0620 03:00:07.946645 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:07.946654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:07.947072 (+ 418us) server_negotiation.cc:407] Connection header received
0620 03:00:07.947198 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:07.947228 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:07.947288 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:07.947372 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:07.948088 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:07.948831 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:07.949966 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:07.950356 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:07.950446 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:07.954116 (+ 3670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38035: BlockingRecv error: recv got EOF from 127.0.97.1:38035 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 03:00:08 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 03:00:08 dist-test-slave-f1j0 krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 03:00:08.412884 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:08.405134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.97.2:51475)
0620 03:00:08.405774 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:08.405790 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:08.405897 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:08.406178 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:08.406181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:08.406559 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:08.406742 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:08.406750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:08.407684 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:08.407687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:08.408599 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:08.408609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:08.409307 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:08.409311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:08.409418 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:08.409976 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:08.409993 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:08.412628 (+ 2635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39051: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":260,"threads_started":1}
W20260620 03:00:08.413050 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:08.405201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51475 (local address 127.0.97.62:39051)
0620 03:00:08.405445 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:08.405451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:08.405858 (+ 407us) server_negotiation.cc:407] Connection header received
0620 03:00:08.405986 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:08.405989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:08.406058 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:08.406140 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:08.406878 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:08.407571 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:08.408718 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:08.409161 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:08.409288 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:08.412822 (+ 3534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51475: BlockingRecv error: recv got EOF from 127.0.97.2:51475 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 03:00:08.740711 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 9215
I20260620 03:00:08.752570 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:08.741888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47026 (local address 127.0.97.1:38859)
0620 03:00:08.742442 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:08.742449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:08.742474 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:00:08.742541 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:08.742547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:08.742777 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:08.742926 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:08.744264 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:08.745304 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:08.746547 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:08.747084 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:08.747195 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:08.749513 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:08.749521 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:08.749535 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:00:08.749619 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:08.752066 (+ 2447us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 03:00:08.752312 (+ 246us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47026: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":265,"threads_started":1}
W20260620 03:00:08.752676 9675 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47026: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 03:00:08.752723 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 9215: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:38859: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 03:00:08.752928 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 9215
I20260620 03:00:08.769678 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 9366
I20260620 03:00:08.782038 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:08.770949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38200 (local address 127.0.97.2:42849)
0620 03:00:08.771549 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:08.771556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:08.771586 (+ 30us) server_negotiation.cc:407] Connection header received
0620 03:00:08.771662 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:08.771669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:08.771881 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:08.772088 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:08.773440 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:08.774594 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:08.775753 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:08.776383 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:08.776481 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:08.779174 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:08.779189 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:08.779203 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:00:08.779238 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:08.781478 (+ 2240us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 03:00:08.781789 (+ 311us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38200: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":270,"threads_started":1}
W20260620 03:00:08.782157 9676 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38200: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 03:00:08.782159 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 9366: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:42849: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 03:00:08.782330 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 9366
I20260620 03:00:08.796065 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 9498
I20260620 03:00:08.807648 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:08.797068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44182 (local address 127.0.97.3:38775)
0620 03:00:08.797701 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:08.797708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:08.797740 (+ 32us) server_negotiation.cc:407] Connection header received
0620 03:00:08.797799 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:08.797805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:08.798027 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:08.798174 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:08.799381 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:08.800405 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:08.801535 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:08.802217 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:08.802315 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:08.804573 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:08.804581 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:08.804594 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:00:08.804627 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:08.807142 (+ 2515us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 03:00:08.807373 (+ 231us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44182: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":325,"threads_started":1}
W20260620 03:00:08.807763 9677 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44182: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 03:00:08.807789 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 9498: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:38775: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 03:00:08.807966 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 9498
I20260620 03:00:08.823958 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 9146
I20260620 03:00:08.825237 9208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:00:08.942179 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 9146
2026-06-20T03:00:08Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21028 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T03:00:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:00:08Z Disabled control of system clock
I20260620 03:00:09.017836 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34449
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 03:00:09.341897 9682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:00:09.342176 9682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:00:09.342253 9682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:00:09.349268 9682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:00:09.349423 9682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:00:09.349489 9682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:00:09.349617 9682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:00:09.349680 9682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:00:09.358778 9682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:00:09.360138 9682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:00:09.362107 9682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:00:09.374346 9687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:09.374441 9690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:09.374452 9688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:00:09.374938 9682 server_base.cc:1034] running on GCE node
I20260620 03:00:09.375734 9682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:00:09.377116 9682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:00:09.378337 9682 hybrid_clock.cc:648] HybridClock initialized: now 1781924409378302 us; error 69 us; skew 500 ppm
I20260620 03:00:09.378681 9682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:00:09.381083 9682 webserver.cc:460] Webserver started at http://127.0.97.62:37335/ using document root <none> and password file <none>
I20260620 03:00:09.381834 9682 fs_manager.cc:359] Metadata directory not provided
I20260620 03:00:09.381963 9682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:00:09.382288 9682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:00:09.385022 9682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "a6320291c9044e599e04ec0e000777f4"
format_stamp: "Formatted at 2026-06-20 03:00:09 on dist-test-slave-f1j0"
server_key: "9849dede975fbbf00f3e08864d7e85c5"
server_key_iv: "aa3c864695eca49299258c9b23466005"
server_key_version: "encryptionkey@0"
I20260620 03:00:09.385881 9682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "a6320291c9044e599e04ec0e000777f4"
format_stamp: "Formatted at 2026-06-20 03:00:09 on dist-test-slave-f1j0"
server_key: "9849dede975fbbf00f3e08864d7e85c5"
server_key_iv: "aa3c864695eca49299258c9b23466005"
server_key_version: "encryptionkey@0"
I20260620 03:00:09.391052 9682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:00:09.394536 9696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:09.395933 9682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 03:00:09.396114 9682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "a6320291c9044e599e04ec0e000777f4"
format_stamp: "Formatted at 2026-06-20 03:00:09 on dist-test-slave-f1j0"
server_key: "9849dede975fbbf00f3e08864d7e85c5"
server_key_iv: "aa3c864695eca49299258c9b23466005"
server_key_version: "encryptionkey@0"
I20260620 03:00:09.396343 9682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:00:09.423789 9682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:00:09.424711 9682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:00:09.425019 9682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:00:09.442795 9682 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34449
I20260620 03:00:09.442816 9747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34449 every 8 connection(s)
I20260620 03:00:09.444152 9682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:00:09.447934 9748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:00:09.451637 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 9682
I20260620 03:00:09.451880 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:00:09.452692 388 external_mini_cluster.cc:1428] Setting key b263f4f4bd7591da251422ac6754afef
I20260620 03:00:09.460669 9748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4: Bootstrap starting.
I20260620 03:00:09.464195 9748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4: Neither blocks nor log segments found. Creating new log.
I20260620 03:00:09.465409 9748 log.cc:824] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4: Log is configured to *not* fsync() on all Append() calls
I20260620 03:00:09.468618 9748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4: No bootstrap required, opened a new log
I20260620 03:00:09.468698 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:09.454771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49560 (local address 127.0.97.62:34449)
0620 03:00:09.455789 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:09.455805 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:09.455859 (+ 54us) server_negotiation.cc:407] Connection header received
0620 03:00:09.458783 (+ 2924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:09.458814 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:09.459408 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:09.459848 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:09.460374 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:09.462301 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:09.463622 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:09.464702 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:09.464877 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:09.466221 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:09.466234 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:09.466260 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:00:09.467017 (+ 757us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:09.467092 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:00:09.467112 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:09.467202 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:09.467429 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 03:00:09.467774 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":299,"threads_started":1}
I20260620 03:00:09.476166 9748 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6320291c9044e599e04ec0e000777f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34449 } }
I20260620 03:00:09.476643 9748 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:00:09.476761 9748 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6320291c9044e599e04ec0e000777f4, State: Initialized, Role: FOLLOWER
I20260620 03:00:09.477391 9748 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6320291c9044e599e04ec0e000777f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34449 } }
I20260620 03:00:09.477662 9748 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:00:09.477800 9748 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:00:09.477982 9748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:00:09.480024 9748 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6320291c9044e599e04ec0e000777f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34449 } }
I20260620 03:00:09.480492 9748 leader_election.cc:302] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6320291c9044e599e04ec0e000777f4; no voters:
I20260620 03:00:09.481036 9748 leader_election.cc:288] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:00:09.481212 9753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:00:09.481922 9753 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4 [term 1 LEADER]: Becoming Leader. State: Replica: a6320291c9044e599e04ec0e000777f4, State: Running, Role: LEADER
I20260620 03:00:09.482568 9753 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6320291c9044e599e04ec0e000777f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34449 } }
I20260620 03:00:09.483214 9748 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:00:09.485360 9755 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6320291c9044e599e04ec0e000777f4. Latest consensus state: current_term: 1 leader_uuid: "a6320291c9044e599e04ec0e000777f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6320291c9044e599e04ec0e000777f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34449 } } }
I20260620 03:00:09.485788 9755 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4 [sys.catalog]: This master's current role is: LEADER
I20260620 03:00:09.485651 9754 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6320291c9044e599e04ec0e000777f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6320291c9044e599e04ec0e000777f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34449 } } }
I20260620 03:00:09.486244 9754 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4 [sys.catalog]: This master's current role is: LEADER
I20260620 03:00:09.488519 9762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:00:09.493978 9762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:00:09.502465 9762 catalog_manager.cc:1347] Generated new cluster ID: fcb09e1d1b154cd0b48298b9e81822d6
I20260620 03:00:09.502575 9762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:00:09.516321 9762 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:00:09.518180 9762 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:00:09.532415 9762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6320291c9044e599e04ec0e000777f4: Generated new TSK 0
I20260620 03:00:09.533556 9762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:00:09.544307 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34449
--builtin_ntp_servers=127.0.97.20:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 03:00:09.873088 9772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:00:09.873390 9772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:00:09.873554 9772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:00:09.880892 9772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:00:09.880993 9772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:00:09.881176 9772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:00:09.890874 9772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:00:09.892279 9772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:00:09.894433 9772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:00:09.907531 9778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:09.907495 9780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:09.907596 9777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:00:09.911026 9772 server_base.cc:1034] running on GCE node
I20260620 03:00:09.911616 9772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:00:09.912602 9772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:00:09.913823 9772 hybrid_clock.cc:648] HybridClock initialized: now 1781924409913796 us; error 53 us; skew 500 ppm
I20260620 03:00:09.914129 9772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:00:09.919276 9772 webserver.cc:460] Webserver started at http://127.0.97.1:41175/ using document root <none> and password file <none>
I20260620 03:00:09.919991 9772 fs_manager.cc:359] Metadata directory not provided
I20260620 03:00:09.920128 9772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:00:09.920455 9772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:00:09.923321 9772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "3c81a7ef1ac64c4e8cbac92365e0e409"
format_stamp: "Formatted at 2026-06-20 03:00:09 on dist-test-slave-f1j0"
server_key: "55baffba33a7be5d84fa408a7de67d3c"
server_key_iv: "ad582c4ff4e5bd0a24356bf8ad160fbf"
server_key_version: "encryptionkey@0"
I20260620 03:00:09.924106 9772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3c81a7ef1ac64c4e8cbac92365e0e409"
format_stamp: "Formatted at 2026-06-20 03:00:09 on dist-test-slave-f1j0"
server_key: "55baffba33a7be5d84fa408a7de67d3c"
server_key_iv: "ad582c4ff4e5bd0a24356bf8ad160fbf"
server_key_version: "encryptionkey@0"
I20260620 03:00:09.930115 9772 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 03:00:09.935590 9786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:09.937718 9772 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.007s sys 0.000s
I20260620 03:00:09.937894 9772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "3c81a7ef1ac64c4e8cbac92365e0e409"
format_stamp: "Formatted at 2026-06-20 03:00:09 on dist-test-slave-f1j0"
server_key: "55baffba33a7be5d84fa408a7de67d3c"
server_key_iv: "ad582c4ff4e5bd0a24356bf8ad160fbf"
server_key_version: "encryptionkey@0"
I20260620 03:00:09.938220 9772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:00:09.985618 9772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:00:09.986553 9772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:00:09.986828 9772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:00:09.987949 9772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:00:09.989511 9772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:00:09.989660 9772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 03:00:09.989789 9772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:00:09.989882 9772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:10.023706 9772 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38749
I20260620 03:00:10.023779 9898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38749 every 8 connection(s)
I20260620 03:00:10.025038 9772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:00:10.026793 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 9772
I20260620 03:00:10.027002 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:00:10.027678 388 external_mini_cluster.cc:1428] Setting key 7f90d590198d9477aed06aa057cc5716
I20260620 03:00:10.030673 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34449
--builtin_ntp_servers=127.0.97.20:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 03:00:10.038583 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:10.027747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51999 (local address 127.0.97.62:34449)
0620 03:00:10.028231 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:10.028238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:10.029748 (+ 1510us) server_negotiation.cc:407] Connection header received
0620 03:00:10.031666 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:10.031674 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:10.031765 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:10.031889 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:10.033024 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:10.034151 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:10.035639 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:10.036364 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:10.036499 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:10.037863 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:10.037867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:10.037870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:00:10.037888 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:10.037924 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:00:10.037936 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:10.038004 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:10.038258 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 03:00:10.038334 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":233,"threads_started":1}
I20260620 03:00:10.040040 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:10.028420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34449 (local address 127.0.97.1:51999)
0620 03:00:10.029547 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:10.029638 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:10.031368 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:10.032148 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:10.032161 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:10.032255 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:10.032869 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:10.032886 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:10.034283 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:10.034287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:10.035506 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:10.035517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:10.036555 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:10.036559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:10.036748 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:10.037467 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:00:10.037472 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:10.037531 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:00:10.037665 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:00:10.038032 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:00:10.038044 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:00:10.038056 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 03:00:10.038249 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 03:00:10.038660 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":330,"threads_started":1}
I20260620 03:00:10.041546 9899 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34449
I20260620 03:00:10.041910 9899 heartbeater.cc:464] Registering TS with master...
I20260620 03:00:10.042644 9899 heartbeater.cc:511] Master 127.0.97.62:34449 requested a full tablet report, sending...
I20260620 03:00:10.044845 9713 ts_manager.cc:194] Registered new tserver with Master: 3c81a7ef1ac64c4e8cbac92365e0e409 (127.0.97.1:38749)
I20260620 03:00:10.046438 9713 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:51999
W20260620 03:00:10.359603 9903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:00:10.359921 9903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:00:10.360086 9903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:00:10.367271 9903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:00:10.367406 9903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:00:10.367584 9903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:00:10.376608 9903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:00:10.377931 9903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:00:10.379719 9903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:00:10.393605 9908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:10.393744 9909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:10.394164 9911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:00:10.395042 9903 server_base.cc:1034] running on GCE node
I20260620 03:00:10.395641 9903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:00:10.396601 9903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:00:10.397794 9903 hybrid_clock.cc:648] HybridClock initialized: now 1781924410397744 us; error 47 us; skew 500 ppm
I20260620 03:00:10.398111 9903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:00:10.400312 9903 webserver.cc:460] Webserver started at http://127.0.97.2:40963/ using document root <none> and password file <none>
I20260620 03:00:10.401012 9903 fs_manager.cc:359] Metadata directory not provided
I20260620 03:00:10.401125 9903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:00:10.401461 9903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:00:10.404268 9903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "ef3310d8e00c4452962c409d1c58e0a8"
format_stamp: "Formatted at 2026-06-20 03:00:10 on dist-test-slave-f1j0"
server_key: "c378d620829ba923b9c8d30fc31da348"
server_key_iv: "9f6ea6b02284f8d4663c5b2e1d3e9581"
server_key_version: "encryptionkey@0"
I20260620 03:00:10.405022 9903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ef3310d8e00c4452962c409d1c58e0a8"
format_stamp: "Formatted at 2026-06-20 03:00:10 on dist-test-slave-f1j0"
server_key: "c378d620829ba923b9c8d30fc31da348"
server_key_iv: "9f6ea6b02284f8d4663c5b2e1d3e9581"
server_key_version: "encryptionkey@0"
I20260620 03:00:10.410142 9903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 03:00:10.413462 9917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:10.414844 9903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:00:10.415007 9903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "ef3310d8e00c4452962c409d1c58e0a8"
format_stamp: "Formatted at 2026-06-20 03:00:10 on dist-test-slave-f1j0"
server_key: "c378d620829ba923b9c8d30fc31da348"
server_key_iv: "9f6ea6b02284f8d4663c5b2e1d3e9581"
server_key_version: "encryptionkey@0"
I20260620 03:00:10.415218 9903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:00:10.428476 9903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:00:10.429394 9903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:00:10.429718 9903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:00:10.430588 9903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:00:10.431962 9903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:00:10.432070 9903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:10.432200 9903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:00:10.432293 9903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:10.464989 9903 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38845
I20260620 03:00:10.465020 10029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38845 every 8 connection(s)
I20260620 03:00:10.466612 9903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:00:10.475260 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 9903
I20260620 03:00:10.475461 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:00:10.475886 388 external_mini_cluster.cc:1428] Setting key e952fc0aa8b1830993e2f925e9378962
I20260620 03:00:10.478557 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34449
--builtin_ntp_servers=127.0.97.20:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 03:00:10.478821 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:10.469201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53287 (local address 127.0.97.62:34449)
0620 03:00:10.469386 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:10.469391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:10.470803 (+ 1412us) server_negotiation.cc:407] Connection header received
0620 03:00:10.472504 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:10.472510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:10.472590 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:10.472713 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:10.473792 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:10.474515 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:10.476179 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:10.476603 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:10.476736 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:10.478064 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:10.478070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:10.478076 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:00:10.478098 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:10.478126 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:00:10.478136 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:10.478227 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:10.478449 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 03:00:10.478537 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 03:00:10.479801 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:10.469749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34449 (local address 127.0.97.2:53287)
0620 03:00:10.470647 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:10.470681 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:10.472243 (+ 1562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:10.472944 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:10.472952 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:10.473036 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:10.473637 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:10.473655 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:10.474659 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:10.474665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:10.476041 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:10.476052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:10.476826 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:10.476834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:10.477055 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:10.477753 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:00:10.477756 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:10.477800 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:00:10.477871 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:00:10.478226 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:00:10.478238 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:00:10.478248 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 03:00:10.478424 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 03:00:10.478800 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":270,"threads_started":1}
I20260620 03:00:10.484258 10030 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34449
I20260620 03:00:10.484578 10030 heartbeater.cc:464] Registering TS with master...
I20260620 03:00:10.485347 10030 heartbeater.cc:511] Master 127.0.97.62:34449 requested a full tablet report, sending...
I20260620 03:00:10.487044 9713 ts_manager.cc:194] Registered new tserver with Master: ef3310d8e00c4452962c409d1c58e0a8 (127.0.97.2:38845)
I20260620 03:00:10.488329 9713 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:53287
W20260620 03:00:10.807135 10033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:00:10.807412 10033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:00:10.807497 10033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:00:10.814635 10033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:00:10.814739 10033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:00:10.814842 10033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:00:10.823961 10033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:00:10.825220 10033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:00:10.826898 10033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:00:10.839210 10038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:10.839200 10039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:10.839257 10041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:00:10.840392 10033 server_base.cc:1034] running on GCE node
I20260620 03:00:10.840966 10033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:00:10.841935 10033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:00:10.843158 10033 hybrid_clock.cc:648] HybridClock initialized: now 1781924410843109 us; error 60 us; skew 500 ppm
I20260620 03:00:10.843467 10033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:00:10.845616 10033 webserver.cc:460] Webserver started at http://127.0.97.3:37175/ using document root <none> and password file <none>
I20260620 03:00:10.846342 10033 fs_manager.cc:359] Metadata directory not provided
I20260620 03:00:10.846474 10033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:00:10.846789 10033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:00:10.849535 10033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "9685ef590a574b78919d6125665318ee"
format_stamp: "Formatted at 2026-06-20 03:00:10 on dist-test-slave-f1j0"
server_key: "fe2cede6b73cb97a4d4c9154dbfd7ab9"
server_key_iv: "1ebba8e5bac6830a4cae17cbbd7437ee"
server_key_version: "encryptionkey@0"
I20260620 03:00:10.850379 10033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9685ef590a574b78919d6125665318ee"
format_stamp: "Formatted at 2026-06-20 03:00:10 on dist-test-slave-f1j0"
server_key: "fe2cede6b73cb97a4d4c9154dbfd7ab9"
server_key_iv: "1ebba8e5bac6830a4cae17cbbd7437ee"
server_key_version: "encryptionkey@0"
I20260620 03:00:10.855259 10033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 03:00:10.858450 10047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:10.859706 10033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:00:10.859885 10033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "9685ef590a574b78919d6125665318ee"
format_stamp: "Formatted at 2026-06-20 03:00:10 on dist-test-slave-f1j0"
server_key: "fe2cede6b73cb97a4d4c9154dbfd7ab9"
server_key_iv: "1ebba8e5bac6830a4cae17cbbd7437ee"
server_key_version: "encryptionkey@0"
I20260620 03:00:10.860095 10033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:00:10.893998 10033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:00:10.894966 10033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:00:10.895260 10033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:00:10.896204 10033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:00:10.897794 10033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:00:10.897913 10033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:10.898052 10033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:00:10.898140 10033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:10.930683 10033 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34747
I20260620 03:00:10.930778 10159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34747 every 8 connection(s)
I20260620 03:00:10.931973 10033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:00:10.934007 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 10033
I20260620 03:00:10.934199 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:00:10.934767 388 external_mini_cluster.cc:1428] Setting key d406c7cc9d1693506766bb7ef1d75093
I20260620 03:00:10.944389 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:10.934673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43901 (local address 127.0.97.62:34449)
0620 03:00:10.934859 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:10.934866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:10.936663 (+ 1797us) server_negotiation.cc:407] Connection header received
0620 03:00:10.938520 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:10.938524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:10.938587 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:10.938667 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:10.939667 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:10.940444 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:10.941975 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:10.942367 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:10.942461 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:10.943633 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:10.943637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:10.943640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:00:10.943657 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:10.943686 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:00:10.943696 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:10.943767 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:10.943974 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 03:00:10.944080 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 03:00:10.945294 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:10.935397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34449 (local address 127.0.97.3:43901)
0620 03:00:10.936485 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:10.936527 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:10.938294 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:10.938856 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:10.938863 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:10.938944 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:10.939493 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:10.939508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:10.940583 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:10.940586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:10.941853 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:10.941864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:10.942538 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:10.942543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:10.942725 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:10.943370 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:00:10.943372 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:10.943408 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:00:10.943476 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:00:10.943781 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:00:10.943788 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:00:10.943793 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 03:00:10.943923 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 03:00:10.944252 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":345,"threads_started":1}
I20260620 03:00:10.946552 10160 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34449
I20260620 03:00:10.946832 10160 heartbeater.cc:464] Registering TS with master...
I20260620 03:00:10.947521 10160 heartbeater.cc:511] Master 127.0.97.62:34449 requested a full tablet report, sending...
I20260620 03:00:10.948899 9713 ts_manager.cc:194] Registered new tserver with Master: 9685ef590a574b78919d6125665318ee (127.0.97.3:34747)
I20260620 03:00:10.949927 9713 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:43901
I20260620 03:00:10.950243 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 03:00:10.956197 9901 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:10.954717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49562 (local address 127.0.97.62:34449)
0620 03:00:10.954892 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:10.954897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:10.955394 (+ 497us) server_negotiation.cc:407] Connection header received
0620 03:00:10.955529 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:10.955532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:10.955602 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:10.955683 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:10.956032 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49562: BlockingRecv error: recv got EOF from 127.0.0.1:49562 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 03:00:10.957000 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 9772
I20260620 03:00:10.964361 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:10.957991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.0.97.1:38749)
0620 03:00:10.958492 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:10.958499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:10.958522 (+ 23us) server_negotiation.cc:407] Connection header received
0620 03:00:10.958573 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:10.958579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:10.958774 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:10.958889 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:10.959197 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:10.960147 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:10.961292 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:10.961905 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:10.961998 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:10.962949 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:10.962956 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:10.962968 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:00:10.963577 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:10.963619 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:00:10.963637 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:10.963763 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:10.963871 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 03:00:10.963969 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":274,"threads_started":1}
I20260620 03:00:10.965476 9894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:00:11.045809 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 9772
I20260620 03:00:11.062819 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 9903
I20260620 03:00:11.070456 10171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:11.063708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.0.97.2:38845)
0620 03:00:11.064293 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:11.064300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:11.064328 (+ 28us) server_negotiation.cc:407] Connection header received
0620 03:00:11.064393 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:11.064399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:11.064614 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:11.064744 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:11.065115 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:11.066174 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:11.067317 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:11.067844 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:11.067938 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:11.069030 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:11.069038 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:11.069051 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:00:11.069696 (+ 645us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:11.069742 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:00:11.069766 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:11.069852 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:11.070028 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 03:00:11.070148 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":323,"threads_started":1}
I20260620 03:00:11.071720 10025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:00:11.154594 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 9903
I20260620 03:00:11.171022 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 10033
I20260620 03:00:11.179023 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:11.172003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60358 (local address 127.0.97.3:34747)
0620 03:00:11.172590 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:11.172597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:11.172626 (+ 29us) server_negotiation.cc:407] Connection header received
0620 03:00:11.172685 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:11.172692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:11.172915 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:11.173041 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:11.173442 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:11.174513 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:11.175762 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:11.176346 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:11.176453 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:11.177417 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:11.177426 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:11.177439 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:00:11.178119 (+ 680us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:11.178188 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:00:11.178209 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:11.178301 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:11.178498 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 03:00:11.178669 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":301,"threads_started":1}
I20260620 03:00:11.180500 10155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:00:11.262918 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 10033
I20260620 03:00:11.279778 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 9682
I20260620 03:00:11.280929 9743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:00:11.381021 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 9682
2026-06-20T03:00:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2433 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T03:00:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:00:11Z Disabled control of system clock
I20260620 03:00:11.451241 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42539
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 03:00:11.775156 10179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:00:11.775460 10179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:00:11.775544 10179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:00:11.782773 10179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:00:11.782881 10179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:00:11.782918 10179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:00:11.782972 10179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:00:11.783016 10179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:00:11.792268 10179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42539
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:00:11.793515 10179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:00:11.795284 10179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:00:11.807370 10184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:11.807406 10187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:11.807337 10185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:00:11.808032 10179 server_base.cc:1034] running on GCE node
I20260620 03:00:11.808768 10179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:00:11.810132 10179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:00:11.811342 10179 hybrid_clock.cc:648] HybridClock initialized: now 1781924411811302 us; error 32 us; skew 500 ppm
I20260620 03:00:11.811695 10179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:00:11.813977 10179 webserver.cc:460] Webserver started at http://127.0.97.62:44847/ using document root <none> and password file <none>
I20260620 03:00:11.814654 10179 fs_manager.cc:359] Metadata directory not provided
I20260620 03:00:11.814775 10179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:00:11.815095 10179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:00:11.818054 10179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "ed0838ed35974c6299e5912e06549bf7"
format_stamp: "Formatted at 2026-06-20 03:00:11 on dist-test-slave-f1j0"
server_key: "86bdb857c175ac0261dc87cdd883511c"
server_key_iv: "c295cc9e0ff57f860ed223c0bd60f92b"
server_key_version: "encryptionkey@0"
I20260620 03:00:11.818861 10179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "ed0838ed35974c6299e5912e06549bf7"
format_stamp: "Formatted at 2026-06-20 03:00:11 on dist-test-slave-f1j0"
server_key: "86bdb857c175ac0261dc87cdd883511c"
server_key_iv: "c295cc9e0ff57f860ed223c0bd60f92b"
server_key_version: "encryptionkey@0"
I20260620 03:00:11.824169 10179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 03:00:11.827783 10193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:11.829221 10179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:00:11.829399 10179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "ed0838ed35974c6299e5912e06549bf7"
format_stamp: "Formatted at 2026-06-20 03:00:11 on dist-test-slave-f1j0"
server_key: "86bdb857c175ac0261dc87cdd883511c"
server_key_iv: "c295cc9e0ff57f860ed223c0bd60f92b"
server_key_version: "encryptionkey@0"
I20260620 03:00:11.829690 10179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:00:11.835208 10179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:00:11.835969 10179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:00:11.836262 10179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:00:11.853881 10179 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42539
I20260620 03:00:11.853911 10244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42539 every 8 connection(s)
I20260620 03:00:11.855211 10179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:00:11.858868 10245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:00:11.862330 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 10179
I20260620 03:00:11.862512 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:00:11.863301 388 external_mini_cluster.cc:1428] Setting key ac97927deb5f86284bf6ade7f2a97b36
I20260620 03:00:11.871536 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:11.865386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.0.97.62:42539)
0620 03:00:11.866394 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:11.866405 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:11.866445 (+ 40us) server_negotiation.cc:407] Connection header received
0620 03:00:11.867993 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:11.868014 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:11.868474 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:11.868888 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:11.869979 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:11.869986 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:11.869998 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:00:11.870011 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:11.870065 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:00:11.870079 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:11.870312 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:11.870446 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 03:00:11.870680 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":364,"threads_started":1}
I20260620 03:00:11.871661 10245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7: Bootstrap starting.
I20260620 03:00:11.875345 10245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7: Neither blocks nor log segments found. Creating new log.
I20260620 03:00:11.876530 10245 log.cc:824] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7: Log is configured to *not* fsync() on all Append() calls
I20260620 03:00:11.879225 10245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7: No bootstrap required, opened a new log
I20260620 03:00:11.886510 10245 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0838ed35974c6299e5912e06549bf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42539 } }
I20260620 03:00:11.886976 10245 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:00:11.887087 10245 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0838ed35974c6299e5912e06549bf7, State: Initialized, Role: FOLLOWER
I20260620 03:00:11.887658 10245 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0838ed35974c6299e5912e06549bf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42539 } }
I20260620 03:00:11.887900 10245 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:00:11.888041 10245 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:00:11.888231 10245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:00:11.890389 10245 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0838ed35974c6299e5912e06549bf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42539 } }
I20260620 03:00:11.890882 10245 leader_election.cc:302] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed0838ed35974c6299e5912e06549bf7; no voters:
I20260620 03:00:11.891445 10245 leader_election.cc:288] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:00:11.891578 10250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:00:11.892292 10250 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7 [term 1 LEADER]: Becoming Leader. State: Replica: ed0838ed35974c6299e5912e06549bf7, State: Running, Role: LEADER
I20260620 03:00:11.892796 10250 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0838ed35974c6299e5912e06549bf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42539 } }
I20260620 03:00:11.893608 10245 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:00:11.895618 10252 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed0838ed35974c6299e5912e06549bf7. Latest consensus state: current_term: 1 leader_uuid: "ed0838ed35974c6299e5912e06549bf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0838ed35974c6299e5912e06549bf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42539 } } }
I20260620 03:00:11.896101 10252 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7 [sys.catalog]: This master's current role is: LEADER
I20260620 03:00:11.895619 10251 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed0838ed35974c6299e5912e06549bf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0838ed35974c6299e5912e06549bf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42539 } } }
I20260620 03:00:11.896764 10251 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7 [sys.catalog]: This master's current role is: LEADER
I20260620 03:00:11.899441 10256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:00:11.904965 10256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:00:11.913683 10256 catalog_manager.cc:1347] Generated new cluster ID: 84156e42ed664693bc4e7bde033d6296
I20260620 03:00:11.913810 10256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:00:11.941676 10256 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:00:11.942179 10256 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:00:11.958092 10256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed0838ed35974c6299e5912e06549bf7: Generated new TSK 0
I20260620 03:00:11.959203 10256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:00:11.978960 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42539
--builtin_ntp_servers=127.0.97.20:33947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 03:00:12.311738 10269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:00:12.312009 10269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:00:12.312095 10269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:00:12.319211 10269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:00:12.319301 10269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:00:12.319406 10269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:00:12.328389 10269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:00:12.329649 10269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:00:12.331277 10269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:00:12.344044 10275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:12.344146 10274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:12.344125 10277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:00:12.345629 10269 server_base.cc:1034] running on GCE node
I20260620 03:00:12.346251 10269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:00:12.347200 10269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:00:12.348409 10269 hybrid_clock.cc:648] HybridClock initialized: now 1781924412348361 us; error 45 us; skew 500 ppm
I20260620 03:00:12.348755 10269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:00:12.350975 10269 webserver.cc:460] Webserver started at http://127.0.97.1:41679/ using document root <none> and password file <none>
I20260620 03:00:12.351712 10269 fs_manager.cc:359] Metadata directory not provided
I20260620 03:00:12.351810 10269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:00:12.352075 10269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:00:12.356359 10269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "c91c862e1b974bd98bbe9b8cd211b1e8"
format_stamp: "Formatted at 2026-06-20 03:00:12 on dist-test-slave-f1j0"
server_key: "600cda1d0a7fdc8fdec8bb4a51e8f855"
server_key_iv: "35621afad48a28abae42bfdb20eae412"
server_key_version: "encryptionkey@0"
I20260620 03:00:12.357131 10269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c91c862e1b974bd98bbe9b8cd211b1e8"
format_stamp: "Formatted at 2026-06-20 03:00:12 on dist-test-slave-f1j0"
server_key: "600cda1d0a7fdc8fdec8bb4a51e8f855"
server_key_iv: "35621afad48a28abae42bfdb20eae412"
server_key_version: "encryptionkey@0"
I20260620 03:00:12.362200 10269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 03:00:12.365423 10283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:12.366778 10269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:00:12.366958 10269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "c91c862e1b974bd98bbe9b8cd211b1e8"
format_stamp: "Formatted at 2026-06-20 03:00:12 on dist-test-slave-f1j0"
server_key: "600cda1d0a7fdc8fdec8bb4a51e8f855"
server_key_iv: "35621afad48a28abae42bfdb20eae412"
server_key_version: "encryptionkey@0"
I20260620 03:00:12.367184 10269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:00:12.372879 10269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:00:12.373672 10269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:00:12.373929 10269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:00:12.374775 10269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:00:12.376062 10269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:00:12.376221 10269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:12.376340 10269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:00:12.376423 10269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:12.408238 10269 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40251
I20260620 03:00:12.408324 10395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40251 every 8 connection(s)
I20260620 03:00:12.409516 10269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:00:12.410454 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 10269
I20260620 03:00:12.410633 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:00:12.411381 388 external_mini_cluster.cc:1428] Setting key 4a26f0372055f6a5f4e291607bc2d27f
I20260620 03:00:12.414235 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42539
--builtin_ntp_servers=127.0.97.20:33947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 03:00:12.418236 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:12.412329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56931 (local address 127.0.97.62:42539)
0620 03:00:12.412777 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:12.412783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:12.414188 (+ 1405us) server_negotiation.cc:407] Connection header received
0620 03:00:12.415862 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:12.415870 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:12.415940 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:12.416053 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:12.417354 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:12.417363 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:12.417370 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:00:12.417392 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:12.417446 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:00:12.417463 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:12.417611 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:12.417810 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 03:00:12.417877 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":237,"threads_started":1}
I20260620 03:00:12.418989 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:12.412953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42539 (local address 127.0.97.1:56931)
0620 03:00:12.414011 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:12.414061 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:12.415597 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:12.416290 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:12.416298 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:12.416397 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:12.417059 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:00:12.417061 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:12.417098 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:00:12.417186 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:00:12.417557 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:00:12.417563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:00:12.417611 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 03:00:12.417837 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 03:00:12.418020 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":344,"threads_started":1}
I20260620 03:00:12.420516 10396 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42539
I20260620 03:00:12.420853 10396 heartbeater.cc:464] Registering TS with master...
I20260620 03:00:12.421423 10396 heartbeater.cc:511] Master 127.0.97.62:42539 requested a full tablet report, sending...
I20260620 03:00:12.423558 10210 ts_manager.cc:194] Registered new tserver with Master: c91c862e1b974bd98bbe9b8cd211b1e8 (127.0.97.1:40251)
W20260620 03:00:12.736989 10400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:00:12.737273 10400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:00:12.737362 10400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:00:12.744773 10400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:00:12.744879 10400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:00:12.744992 10400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:00:12.754146 10400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:00:12.755415 10400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:00:12.757097 10400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:00:12.770412 10406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:12.770653 10405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:12.771020 10408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:00:12.771344 10400 server_base.cc:1034] running on GCE node
I20260620 03:00:12.772114 10400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:00:12.773198 10400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:00:12.774463 10400 hybrid_clock.cc:648] HybridClock initialized: now 1781924412774422 us; error 69 us; skew 500 ppm
I20260620 03:00:12.774819 10400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:00:12.777227 10400 webserver.cc:460] Webserver started at http://127.0.97.2:40763/ using document root <none> and password file <none>
I20260620 03:00:12.777994 10400 fs_manager.cc:359] Metadata directory not provided
I20260620 03:00:12.778117 10400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:00:12.778434 10400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:00:12.781260 10400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "f20adaae1084455b94029defe3c9454a"
format_stamp: "Formatted at 2026-06-20 03:00:12 on dist-test-slave-f1j0"
server_key: "9c10d55f8ba33370db9e2372405337fc"
server_key_iv: "f5edd6990c6c8a7affdba97af44ddc87"
server_key_version: "encryptionkey@0"
I20260620 03:00:12.782038 10400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f20adaae1084455b94029defe3c9454a"
format_stamp: "Formatted at 2026-06-20 03:00:12 on dist-test-slave-f1j0"
server_key: "9c10d55f8ba33370db9e2372405337fc"
server_key_iv: "f5edd6990c6c8a7affdba97af44ddc87"
server_key_version: "encryptionkey@0"
I20260620 03:00:12.787056 10400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 03:00:12.790436 10414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:12.791757 10400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 03:00:12.791934 10400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "f20adaae1084455b94029defe3c9454a"
format_stamp: "Formatted at 2026-06-20 03:00:12 on dist-test-slave-f1j0"
server_key: "9c10d55f8ba33370db9e2372405337fc"
server_key_iv: "f5edd6990c6c8a7affdba97af44ddc87"
server_key_version: "encryptionkey@0"
I20260620 03:00:12.792183 10400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:00:12.797940 10400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:00:12.798755 10400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:00:12.799010 10400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:00:12.799913 10400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:00:12.801321 10400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:00:12.801436 10400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:12.801553 10400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:00:12.801746 10400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 03:00:12.833789 10400 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45995
I20260620 03:00:12.833839 10526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45995 every 8 connection(s)
I20260620 03:00:12.835173 10400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:00:12.837286 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 10400
I20260620 03:00:12.837414 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:00:12.838042 388 external_mini_cluster.cc:1428] Setting key b63aff75a189195af1b409586a791dd6
I20260620 03:00:12.840541 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42539
--builtin_ntp_servers=127.0.97.20:33947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 03:00:12.844023 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:12.837971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46901 (local address 127.0.97.62:42539)
0620 03:00:12.838177 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:12.838183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:12.839846 (+ 1663us) server_negotiation.cc:407] Connection header received
0620 03:00:12.841669 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:12.841678 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:12.841758 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:12.841871 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:12.843236 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:12.843244 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:12.843252 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:00:12.843275 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:12.843315 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:00:12.843332 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:12.843450 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:12.843610 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 03:00:12.843676 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:00:12.844878 10529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:12.838588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42539 (local address 127.0.97.2:46901)
0620 03:00:12.839686 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:12.839730 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:12.841390 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:12.842079 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:12.842088 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:12.842182 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:12.842942 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:00:12.842945 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:12.842983 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:00:12.843068 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:00:12.843427 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:00:12.843434 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:00:12.843439 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 03:00:12.843647 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 03:00:12.843889 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":376,"threads_started":1}
I20260620 03:00:12.846261 10527 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42539
I20260620 03:00:12.846567 10527 heartbeater.cc:464] Registering TS with master...
I20260620 03:00:12.847180 10527 heartbeater.cc:511] Master 127.0.97.62:42539 requested a full tablet report, sending...
I20260620 03:00:12.848555 10210 ts_manager.cc:194] Registered new tserver with Master: f20adaae1084455b94029defe3c9454a (127.0.97.2:45995)
W20260620 03:00:13.167292 10530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:00:13.167595 10530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:00:13.167754 10530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:00:13.175292 10530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:00:13.175422 10530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:00:13.175595 10530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:00:13.184667 10530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:00:13.186079 10530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:00:13.187865 10530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:00:13.200932 10535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:13.201129 10538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:13.201170 10536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:00:13.202144 10530 server_base.cc:1034] running on GCE node
I20260620 03:00:13.202718 10530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:00:13.203634 10530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:00:13.204860 10530 hybrid_clock.cc:648] HybridClock initialized: now 1781924413204799 us; error 54 us; skew 500 ppm
I20260620 03:00:13.205171 10530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:00:13.207576 10530 webserver.cc:460] Webserver started at http://127.0.97.3:40085/ using document root <none> and password file <none>
I20260620 03:00:13.208391 10530 fs_manager.cc:359] Metadata directory not provided
I20260620 03:00:13.208506 10530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:00:13.208829 10530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:00:13.211583 10530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "2aa525613be84241a35af75c1862a5b7"
format_stamp: "Formatted at 2026-06-20 03:00:13 on dist-test-slave-f1j0"
server_key: "344e844f5fe30be4b8edf4b1b1107235"
server_key_iv: "3b85b673fb8f1d61cec7920faa017003"
server_key_version: "encryptionkey@0"
I20260620 03:00:13.212412 10530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2aa525613be84241a35af75c1862a5b7"
format_stamp: "Formatted at 2026-06-20 03:00:13 on dist-test-slave-f1j0"
server_key: "344e844f5fe30be4b8edf4b1b1107235"
server_key_iv: "3b85b673fb8f1d61cec7920faa017003"
server_key_version: "encryptionkey@0"
I20260620 03:00:13.217267 10530 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 03:00:13.220499 10544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:13.221845 10530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:00:13.221997 10530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "2aa525613be84241a35af75c1862a5b7"
format_stamp: "Formatted at 2026-06-20 03:00:13 on dist-test-slave-f1j0"
server_key: "344e844f5fe30be4b8edf4b1b1107235"
server_key_iv: "3b85b673fb8f1d61cec7920faa017003"
server_key_version: "encryptionkey@0"
I20260620 03:00:13.222182 10530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:00:13.227696 10530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:00:13.228457 10530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:00:13.228720 10530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:00:13.229696 10530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:00:13.231132 10530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:00:13.231213 10530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:13.231376 10530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:00:13.231457 10530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:13.264462 10530 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41445
I20260620 03:00:13.264514 10656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41445 every 8 connection(s)
I20260620 03:00:13.265812 10530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:00:13.273736 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 10530
I20260620 03:00:13.273902 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:00:13.274418 388 external_mini_cluster.cc:1428] Setting key 1e64ae6575c921ce92c7de9b9b3a581f
I20260620 03:00:13.274399 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:13.268321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47255 (local address 127.0.97.62:42539)
0620 03:00:13.268525 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:13.268531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:13.269981 (+ 1450us) server_negotiation.cc:407] Connection header received
0620 03:00:13.271671 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:13.271675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:13.271725 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:13.271833 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:13.273228 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:13.273235 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:13.273241 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:00:13.273278 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:13.273317 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:00:13.273333 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:13.273451 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:13.273830 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 03:00:13.273942 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 03:00:13.275318 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:13.268902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42539 (local address 127.0.97.3:47255)
0620 03:00:13.269833 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:13.269870 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:13.271439 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:13.272030 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:13.272037 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:13.272125 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:13.272925 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:00:13.272928 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:13.272975 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:00:13.273058 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:00:13.273525 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:00:13.273536 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:00:13.273546 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 03:00:13.273866 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 03:00:13.274125 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":274,"threads_started":1}
I20260620 03:00:13.276997 10657 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42539
I20260620 03:00:13.277319 10657 heartbeater.cc:464] Registering TS with master...
I20260620 03:00:13.277964 10657 heartbeater.cc:511] Master 127.0.97.62:42539 requested a full tablet report, sending...
I20260620 03:00:13.279280 10209 ts_manager.cc:194] Registered new tserver with Master: 2aa525613be84241a35af75c1862a5b7 (127.0.97.3:41445)
I20260620 03:00:13.289479 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 03:00:13.295619 10398 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:00:13.294070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54522 (local address 127.0.97.62:42539)
0620 03:00:13.294229 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:13.294233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:13.294751 (+ 518us) server_negotiation.cc:407] Connection header received
0620 03:00:13.294881 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:13.294885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:13.294937 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:13.295058 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:13.295369 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54522: BlockingRecv error: recv got EOF from 127.0.0.1:54522 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 03:00:13.296132 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 10269
I20260620 03:00:13.300143 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:13.297509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39796 (local address 127.0.97.1:40251)
0620 03:00:13.298065 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:13.298078 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:13.298103 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:00:13.298163 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:13.298168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:13.298382 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:13.298510 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:13.299449 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:13.299457 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:13.299469 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:00:13.299503 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:13.299592 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:00:13.299619 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:13.299762 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:13.299833 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 03:00:13.299912 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":284,"threads_started":1}
I20260620 03:00:13.301441 10391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:00:13.382354 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 10269
I20260620 03:00:13.400662 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 10400
I20260620 03:00:13.404449 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:13.401801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.0.97.2:45995)
0620 03:00:13.402370 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:13.402384 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:13.402411 (+ 27us) server_negotiation.cc:407] Connection header received
0620 03:00:13.402473 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:13.402479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:13.402701 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:13.402831 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:13.403745 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:13.403751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:13.403763 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:00:13.403780 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:13.403818 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:00:13.403832 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:13.403937 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:13.404017 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 03:00:13.404128 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":277,"threads_started":1}
I20260620 03:00:13.405853 10522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:00:13.487195 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 10400
I20260620 03:00:13.504415 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 10530
I20260620 03:00:13.508512 10670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:13.505437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58132 (local address 127.0.97.3:41445)
0620 03:00:13.506100 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:13.506106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:13.506137 (+ 31us) server_negotiation.cc:407] Connection header received
0620 03:00:13.506197 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:13.506204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:13.506443 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:13.506613 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:13.507756 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:13.507762 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:13.507773 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:00:13.507790 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:13.507826 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:00:13.507839 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:13.508015 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:13.508103 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 03:00:13.508214 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":373,"threads_started":1}
I20260620 03:00:13.509778 10652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:00:13.589444 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 10530
I20260620 03:00:13.605377 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 10179
I20260620 03:00:13.606658 10240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:00:13.706341 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 10179
2026-06-20T03:00:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2324 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:00:13 dist-test-slave-f1j0 krb5kdc[10675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:00:13 dist-test-slave-f1j0 krb5kdc[10675](info): set up 2 sockets
Jun 20 03:00:13 dist-test-slave-f1j0 krb5kdc[10675](info): commencing operation
krb5kdc: starting...
W20260620 03:00:14.798632 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:00:14 dist-test-slave-f1j0 krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:00:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:00:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:00:15.019165 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33043
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:00:15.339690 10688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:00:15.339967 10688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:00:15.340140 10688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:00:15.347193 10688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:00:15.347312 10688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:00:15.347433 10688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:00:15.347536 10688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:00:15.347612 10688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:00:15.357276 10688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:00:15.358642 10688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:00:15.359922 10688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:00:15.371376 10694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:15.371433 10696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:15.371349 10693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:00:15.372135 10688 server_base.cc:1034] running on GCE node
I20260620 03:00:15.373041 10688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:00:15.374531 10688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:00:15.375741 10688 hybrid_clock.cc:648] HybridClock initialized: now 1781924415375699 us; error 35 us; skew 500 ppm
I20260620 03:00:15.376093 10688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:00:15 dist-test-slave-f1j0 krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:00:15.379096 10688 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:00:15.380587 10688 webserver.cc:460] Webserver started at http://127.0.97.62:40315/ using document root <none> and password file <none>
I20260620 03:00:15.381276 10688 fs_manager.cc:359] Metadata directory not provided
I20260620 03:00:15.381433 10688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:00:15.381803 10688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:00:15.384738 10688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "bc609528baf04c9bb702d600058ba37d"
format_stamp: "Formatted at 2026-06-20 03:00:15 on dist-test-slave-f1j0"
server_key: "937d4b6f6a37996c64a055b9af028344"
server_key_iv: "1f40ea6081798f14aef559d3f499cd70"
server_key_version: "encryptionkey@0"
I20260620 03:00:15.385502 10688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "bc609528baf04c9bb702d600058ba37d"
format_stamp: "Formatted at 2026-06-20 03:00:15 on dist-test-slave-f1j0"
server_key: "937d4b6f6a37996c64a055b9af028344"
server_key_iv: "1f40ea6081798f14aef559d3f499cd70"
server_key_version: "encryptionkey@0"
I20260620 03:00:15.390637 10688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:00:15.394083 10703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:15.395403 10688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 03:00:15.395583 10688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "bc609528baf04c9bb702d600058ba37d"
format_stamp: "Formatted at 2026-06-20 03:00:15 on dist-test-slave-f1j0"
server_key: "937d4b6f6a37996c64a055b9af028344"
server_key_iv: "1f40ea6081798f14aef559d3f499cd70"
server_key_version: "encryptionkey@0"
I20260620 03:00:15.395777 10688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:00:15.413774 10688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:00:15.417721 10688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:00:15.418010 10688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:00:15.436111 10688 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33043
I20260620 03:00:15.436143 10754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33043 every 8 connection(s)
I20260620 03:00:15.437426 10688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:00:15.440295 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 10688
I20260620 03:00:15.440495 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:00:15.441287 388 external_mini_cluster.cc:1428] Setting key b9576145401db3464e8a7f938528a96e
I20260620 03:00:15.441486 10755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:00:15.453931 10755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d: Bootstrap starting.
Jun 20 03:00:15 dist-test-slave-f1j0 krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:00:15.457316 10755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d: Neither blocks nor log segments found. Creating new log.
I20260620 03:00:15.458534 10755 log.cc:824] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d: Log is configured to *not* fsync() on all Append() calls
I20260620 03:00:15.461953 10755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d: No bootstrap required, opened a new log
I20260620 03:00:15.463757 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:15.443379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60196 (local address 127.0.97.62:33043)
0620 03:00:15.444110 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:15.444122 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:15.444160 (+ 38us) server_negotiation.cc:407] Connection header received
0620 03:00:15.446258 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:15.446281 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:15.446681 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:15.447093 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:15.448534 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:15.449846 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:15.451122 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:15.451822 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:15.452058 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:15.456195 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:15.456203 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:15.456217 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:00:15.456250 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:15.459772 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:00:15.460582 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:00:15.460588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:00:15.460594 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:00:15.460744 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:00:15.461137 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:00:15.461140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:00:15.461142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:00:15.461703 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:15.461927 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:15.462364 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 03:00:15.462732 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":323,"threads_started":1}
I20260620 03:00:15.469964 10755 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc609528baf04c9bb702d600058ba37d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33043 } }
I20260620 03:00:15.470322 10755 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:00:15.470439 10755 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc609528baf04c9bb702d600058ba37d, State: Initialized, Role: FOLLOWER
I20260620 03:00:15.470970 10755 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc609528baf04c9bb702d600058ba37d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33043 } }
I20260620 03:00:15.471190 10755 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:00:15.471336 10755 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:00:15.471503 10755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:00:15.473559 10755 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc609528baf04c9bb702d600058ba37d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33043 } }
I20260620 03:00:15.474086 10755 leader_election.cc:302] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc609528baf04c9bb702d600058ba37d; no voters:
I20260620 03:00:15.474674 10755 leader_election.cc:288] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:00:15.474823 10760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:00:15.475847 10760 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d [term 1 LEADER]: Becoming Leader. State: Replica: bc609528baf04c9bb702d600058ba37d, State: Running, Role: LEADER
I20260620 03:00:15.476354 10760 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc609528baf04c9bb702d600058ba37d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33043 } }
I20260620 03:00:15.476826 10755 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:00:15.479646 10761 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc609528baf04c9bb702d600058ba37d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc609528baf04c9bb702d600058ba37d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33043 } } }
I20260620 03:00:15.479619 10762 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc609528baf04c9bb702d600058ba37d. Latest consensus state: current_term: 1 leader_uuid: "bc609528baf04c9bb702d600058ba37d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc609528baf04c9bb702d600058ba37d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33043 } } }
I20260620 03:00:15.480014 10761 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d [sys.catalog]: This master's current role is: LEADER
I20260620 03:00:15.480059 10762 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d [sys.catalog]: This master's current role is: LEADER
I20260620 03:00:15.481397 10765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:00:15.486228 10765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:00:15.494781 10765 catalog_manager.cc:1347] Generated new cluster ID: 9636592e70a5444bb32f95c259ae2ed7
I20260620 03:00:15.494912 10765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:00:15.521322 10765 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:00:15.523062 10765 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:00:15.530460 10765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d: Generated new TSK 0
I20260620 03:00:15.531401 10765 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:00:15.601984 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33043
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:00:15.932307 10781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:00:15.932595 10781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:00:15.932770 10781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:00:15.939894 10781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:00:15.940003 10781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:00:15.940219 10781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:00:15.949980 10781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:00:15.951314 10781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:00:15.952631 10781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:00:15.966015 10787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:15.966118 10786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:15.966092 10789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:00:15.966918 10781 server_base.cc:1034] running on GCE node
I20260620 03:00:15.967517 10781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:00:15.968493 10781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:00:15.969698 10781 hybrid_clock.cc:648] HybridClock initialized: now 1781924415969635 us; error 50 us; skew 500 ppm
I20260620 03:00:15.970016 10781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:00:15 dist-test-slave-f1j0 krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:00:15.975768 10781 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:00:15.977424 10781 webserver.cc:460] Webserver started at http://127.0.97.1:34277/ using document root <none> and password file <none>
I20260620 03:00:15.978245 10781 fs_manager.cc:359] Metadata directory not provided
I20260620 03:00:15.978387 10781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:00:15.978689 10781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:00:15.981843 10781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "f17443026f2943059043ec77f4ce5123"
format_stamp: "Formatted at 2026-06-20 03:00:15 on dist-test-slave-f1j0"
server_key: "f12f47c0b8a70edc9317f463f60963c8"
server_key_iv: "6cfc058681b3356fb0bf0620d6ea3acb"
server_key_version: "encryptionkey@0"
I20260620 03:00:15.982669 10781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f17443026f2943059043ec77f4ce5123"
format_stamp: "Formatted at 2026-06-20 03:00:15 on dist-test-slave-f1j0"
server_key: "f12f47c0b8a70edc9317f463f60963c8"
server_key_iv: "6cfc058681b3356fb0bf0620d6ea3acb"
server_key_version: "encryptionkey@0"
I20260620 03:00:15.987864 10781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 03:00:15.991357 10796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:15.992944 10781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:00:15.993120 10781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "f17443026f2943059043ec77f4ce5123"
format_stamp: "Formatted at 2026-06-20 03:00:15 on dist-test-slave-f1j0"
server_key: "f12f47c0b8a70edc9317f463f60963c8"
server_key_iv: "6cfc058681b3356fb0bf0620d6ea3acb"
server_key_version: "encryptionkey@0"
I20260620 03:00:15.993319 10781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:00:16.017146 10781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:00:16.021768 10781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:00:16.022084 10781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:00:16.023053 10781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:00:16.024394 10781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:00:16.024466 10781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:16.024627 10781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:00:16.024691 10781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:16.055802 10781 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44593
I20260620 03:00:16.055855 10908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44593 every 8 connection(s)
I20260620 03:00:16.057096 10781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:00:16.067519 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 10781
I20260620 03:00:16.067775 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:00:16.068535 388 external_mini_cluster.cc:1428] Setting key db056dea928d24f6b93dde49dc2349e2
Jun 20 03:00:16 dist-test-slave-f1j0 krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:00:16.077271 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:16.059675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53183 (local address 127.0.97.62:33043)
0620 03:00:16.060115 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:16.060119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:16.061112 (+ 993us) server_negotiation.cc:407] Connection header received
0620 03:00:16.062922 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:16.062926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:16.062998 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:16.063096 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:16.064533 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:16.065240 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:16.066784 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:16.067327 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:16.067471 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:16.071362 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:16.071367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:16.071370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:00:16.071399 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:16.074002 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:00:16.074906 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:00:16.074910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:00:16.074912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:00:16.075019 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:00:16.075498 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:00:16.075501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:00:16.075502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:00:16.075737 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:16.075859 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:16.076782 (+ 923us) server_negotiation.cc:299] Negotiation successful
0620 03:00:16.076980 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":219,"threads_started":1}
I20260620 03:00:16.078485 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:16.060255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33043 (local address 127.0.97.1:53183)
0620 03:00:16.060952 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:16.060990 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:16.062685 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:16.063318 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:16.063326 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:16.063809 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:16.064380 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:16.064397 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:16.065374 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:16.065377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:16.066659 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:16.066670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:16.067566 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:16.067572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:16.067887 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:16.068731 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:16.068754 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:16.071146 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:00:16.074143 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:00:16.074155 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:00:16.074175 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:00:16.074792 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:00:16.075120 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:00:16.075127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:00:16.075133 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:00:16.075391 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:00:16.075864 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:00:16.075875 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:00:16.076344 (+ 469us) client_negotiation.cc:769] Sending connection context
0620 03:00:16.076653 (+ 309us) client_negotiation.cc:240] Negotiation successful
0620 03:00:16.077272 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":275,"threads_started":1}
I20260620 03:00:16.079993 10909 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33043
I20260620 03:00:16.080317 10909 heartbeater.cc:464] Registering TS with master...
I20260620 03:00:16.081197 10909 heartbeater.cc:511] Master 127.0.97.62:33043 requested a full tablet report, sending...
I20260620 03:00:16.083566 10720 ts_manager.cc:194] Registered new tserver with Master: f17443026f2943059043ec77f4ce5123 (127.0.97.1:44593)
I20260620 03:00:16.085065 10720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53183
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:00:16.120716 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33043
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:00:16.449385 10915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:00:16.449759 10915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:00:16.449934 10915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:00:16.457257 10915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:00:16.457419 10915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:00:16.457638 10915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:00:16.467474 10915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:00:16.468832 10915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:00:16.470418 10915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:00:16.483819 10920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:16.483778 10923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:16.483857 10921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:00:16.485025 10915 server_base.cc:1034] running on GCE node
I20260620 03:00:16.485781 10915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:00:16.486737 10915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:00:16.487933 10915 hybrid_clock.cc:648] HybridClock initialized: now 1781924416487873 us; error 47 us; skew 500 ppm
I20260620 03:00:16.488262 10915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:00:16 dist-test-slave-f1j0 krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:00:16.491470 10915 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:00:16.492966 10915 webserver.cc:460] Webserver started at http://127.0.97.2:39505/ using document root <none> and password file <none>
I20260620 03:00:16.493721 10915 fs_manager.cc:359] Metadata directory not provided
I20260620 03:00:16.493826 10915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:00:16.494079 10915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:00:16.497665 10915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "9f0fa42927ee437f82ae1da8c0ac94c6"
format_stamp: "Formatted at 2026-06-20 03:00:16 on dist-test-slave-f1j0"
server_key: "f386e2316251b4730dc59f387b041404"
server_key_iv: "610192475597aedee56652cf05d0de83"
server_key_version: "encryptionkey@0"
I20260620 03:00:16.498435 10915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9f0fa42927ee437f82ae1da8c0ac94c6"
format_stamp: "Formatted at 2026-06-20 03:00:16 on dist-test-slave-f1j0"
server_key: "f386e2316251b4730dc59f387b041404"
server_key_iv: "610192475597aedee56652cf05d0de83"
server_key_version: "encryptionkey@0"
I20260620 03:00:16.504045 10915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:00:16.507583 10930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:16.509086 10915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:00:16.509244 10915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "9f0fa42927ee437f82ae1da8c0ac94c6"
format_stamp: "Formatted at 2026-06-20 03:00:16 on dist-test-slave-f1j0"
server_key: "f386e2316251b4730dc59f387b041404"
server_key_iv: "610192475597aedee56652cf05d0de83"
server_key_version: "encryptionkey@0"
I20260620 03:00:16.509444 10915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:00:16.531051 10915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:00:16.535532 10915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:00:16.535827 10915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:00:16.536784 10915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:00:16.538118 10915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:00:16.538193 10915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:16.538365 10915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:00:16.538414 10915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:16.571075 10915 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37675
I20260620 03:00:16.571208 11042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37675 every 8 connection(s)
I20260620 03:00:16.572333 10915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:00:16.577354 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 10915
I20260620 03:00:16.577522 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:00:16.578032 388 external_mini_cluster.cc:1428] Setting key d9acc81b487b9e5927efb512512e3e2e
Jun 20 03:00:16 dist-test-slave-f1j0 krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:00:16.593425 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:16.574932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35765 (local address 127.0.97.62:33043)
0620 03:00:16.575141 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:16.575147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:16.576320 (+ 1173us) server_negotiation.cc:407] Connection header received
0620 03:00:16.578383 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:16.578389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:16.578478 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:16.578598 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:16.580508 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:16.581646 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:16.583270 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:16.584038 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:16.584175 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:16.587438 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:16.587447 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:16.587454 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:00:16.587496 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:16.590342 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:00:16.591240 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:00:16.591244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:00:16.591246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:00:16.591355 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:00:16.591833 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:00:16.591836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:00:16.591838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:00:16.592087 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:16.592189 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:16.593038 (+ 849us) server_negotiation.cc:299] Negotiation successful
0620 03:00:16.593191 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 03:00:16.595121 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:16.575457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33043 (local address 127.0.97.2:35765)
0620 03:00:16.576150 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:16.576192 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:16.578051 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:16.578887 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:16.578904 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:16.579577 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:16.580341 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:16.580359 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:16.581779 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:16.581783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:16.583144 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:16.583155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:16.584229 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:16.584234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:16.584421 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:16.585080 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:16.585105 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:16.587209 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:00:16.590478 (+ 3269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:00:16.590490 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:00:16.590510 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:00:16.591128 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:00:16.591459 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:00:16.591467 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:00:16.591474 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:00:16.591728 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:00:16.592208 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:00:16.592218 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:00:16.592646 (+ 428us) client_negotiation.cc:769] Sending connection context
0620 03:00:16.592968 (+ 322us) client_negotiation.cc:240] Negotiation successful
0620 03:00:16.593528 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":276,"threads_started":1}
I20260620 03:00:16.596570 11043 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33043
I20260620 03:00:16.596923 11043 heartbeater.cc:464] Registering TS with master...
I20260620 03:00:16.597854 11043 heartbeater.cc:511] Master 127.0.97.62:33043 requested a full tablet report, sending...
I20260620 03:00:16.599498 10720 ts_manager.cc:194] Registered new tserver with Master: 9f0fa42927ee437f82ae1da8c0ac94c6 (127.0.97.2:37675)
I20260620 03:00:16.600464 10720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35765
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:00:16.632670 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33043
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:00:16.956095 11048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:00:16.956389 11048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:00:16.956552 11048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:00:16.963827 11048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:00:16.963932 11048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:00:16.964147 11048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:00:16.973891 11048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:00:16.975162 11048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:00:16.976449 11048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:00:16.990043 11053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:16.989977 11054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:00:16.990571 11056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:00:16.991051 11048 server_base.cc:1034] running on GCE node
I20260620 03:00:16.991657 11048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:00:16.992611 11048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:00:16.993842 11048 hybrid_clock.cc:648] HybridClock initialized: now 1781924416993791 us; error 44 us; skew 500 ppm
I20260620 03:00:16.994159 11048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:00:16 dist-test-slave-f1j0 krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:00:16.997284 11048 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:00:16.998997 11048 webserver.cc:460] Webserver started at http://127.0.97.3:33569/ using document root <none> and password file <none>
I20260620 03:00:16.999709 11048 fs_manager.cc:359] Metadata directory not provided
I20260620 03:00:16.999858 11048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:00:17.000204 11048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:00:17.003201 11048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "d3f0dd85846b4c27801cf00f85954eb6"
format_stamp: "Formatted at 2026-06-20 03:00:16 on dist-test-slave-f1j0"
server_key: "4dd516966451bfc4952d0b40d262a02a"
server_key_iv: "7aa695f437773ba2027b7e298d58c6c2"
server_key_version: "encryptionkey@0"
I20260620 03:00:17.003981 11048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d3f0dd85846b4c27801cf00f85954eb6"
format_stamp: "Formatted at 2026-06-20 03:00:16 on dist-test-slave-f1j0"
server_key: "4dd516966451bfc4952d0b40d262a02a"
server_key_iv: "7aa695f437773ba2027b7e298d58c6c2"
server_key_version: "encryptionkey@0"
I20260620 03:00:17.009151 11048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:00:17.012621 11063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:17.014132 11048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:00:17.014300 11048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "d3f0dd85846b4c27801cf00f85954eb6"
format_stamp: "Formatted at 2026-06-20 03:00:16 on dist-test-slave-f1j0"
server_key: "4dd516966451bfc4952d0b40d262a02a"
server_key_iv: "7aa695f437773ba2027b7e298d58c6c2"
server_key_version: "encryptionkey@0"
I20260620 03:00:17.014499 11048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:00:17.038230 11048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:00:17.041996 11048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:00:17.042273 11048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:00:17.043171 11048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:00:17.044426 11048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:00:17.044526 11048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:17.044672 11048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:00:17.044747 11048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:00:17.077438 11048 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34579
I20260620 03:00:17.077525 11175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34579 every 8 connection(s)
I20260620 03:00:17.078763 11048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:00:17.084919 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 11048
I20260620 03:00:17.085109 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:00:17.085700 388 external_mini_cluster.cc:1428] Setting key 67ff3cbc4e7b95eebf07216af8488a00
I20260620 03:00:17.089001 10909 heartbeater.cc:503] Master 127.0.97.62:33043 was elected leader, sending a full tablet report...
Jun 20 03:00:17 dist-test-slave-f1j0 krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:00:17.098100 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:17.081151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35055 (local address 127.0.97.62:33043)
0620 03:00:17.081369 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:17.081376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:17.082578 (+ 1202us) server_negotiation.cc:407] Connection header received
0620 03:00:17.084338 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:17.084342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:17.084410 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:17.084496 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:17.086482 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.087446 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.089073 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.089595 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.089715 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:17.093081 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:17.093086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:17.093090 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:00:17.093120 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:17.095498 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:00:17.096194 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:00:17.096197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:00:17.096200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:00:17.096322 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:00:17.096667 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:00:17.096669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:00:17.096671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:00:17.096914 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:17.097024 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:17.097710 (+ 686us) server_negotiation.cc:299] Negotiation successful
0620 03:00:17.097870 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 03:00:17.099164 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:17.081739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33043 (local address 127.0.97.3:35055)
0620 03:00:17.082407 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:17.082449 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:17.084082 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:17.084810 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:17.084823 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:17.085531 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:00:17.086322 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:17.086341 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.087587 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.087596 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:17.088932 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:17.088946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.089802 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.089811 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:17.090040 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:17.090766 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:00:17.090788 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:00:17.092883 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:00:17.095653 (+ 2770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:00:17.095660 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:00:17.095673 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:00:17.096080 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:00:17.096414 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:00:17.096418 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:00:17.096420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:00:17.096580 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:00:17.097024 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:00:17.097030 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:00:17.097341 (+ 311us) client_negotiation.cc:769] Sending connection context
0620 03:00:17.097560 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 03:00:17.098011 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":261,"threads_started":1}
I20260620 03:00:17.100570 11176 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33043
I20260620 03:00:17.100884 11176 heartbeater.cc:464] Registering TS with master...
I20260620 03:00:17.101559 11176 heartbeater.cc:511] Master 127.0.97.62:33043 requested a full tablet report, sending...
I20260620 03:00:17.102985 10720 ts_manager.cc:194] Registered new tserver with Master: d3f0dd85846b4c27801cf00f85954eb6 (127.0.97.3:34579)
I20260620 03:00:17.103760 10720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35055
I20260620 03:00:17.114830 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:00:17.130982 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:17.119270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60200 (local address 127.0.97.62:33043)
0620 03:00:17.119434 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:17.119439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:17.119884 (+ 445us) server_negotiation.cc:407] Connection header received
0620 03:00:17.119999 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:17.120002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:17.120063 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:17.120138 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:17.121367 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.122125 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.123317 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.123711 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.123809 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:17.125949 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:17.125953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:17.125956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:00:17.125986 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:17.128512 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:00:17.129088 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:00:17.129091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:00:17.129094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:00:17.129203 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:00:17.129512 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:00:17.129515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:00:17.129517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:00:17.129860 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:17.130033 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:17.130499 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 03:00:17.130712 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 03:00:17.135533 10720 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:00:17.142637 10720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:00:17.167737 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:17.154537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37675 (local address 127.0.0.1:37416)
0620 03:00:17.155939 (+ 1402us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:17.155975 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:17.156127 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:17.160337 (+ 4210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:17.160345 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:17.160381 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:00:17.161152 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:17.161169 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.163103 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.163110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:17.164742 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:17.164767 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.166647 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.166657 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:17.166907 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:17.166966 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 03:00:17.167082 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 03:00:17.167264 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1170,"mutex_wait_us":3,"spinlock_wait_cycles":6400,"thread_start_us":4108,"threads_started":1}
I20260620 03:00:17.168123 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:17.154757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44593 (local address 127.0.0.1:36952)
0620 03:00:17.158943 (+ 4186us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:17.158964 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:17.159165 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:17.160340 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:17.160348 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:17.160416 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:00:17.161318 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:17.161334 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.163147 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.163158 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:17.165279 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:17.165292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.167081 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.167087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:17.167240 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:17.167286 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 03:00:17.167401 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 03:00:17.167524 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4038,"mutex_wait_us":81,"spinlock_wait_cycles":23936,"thread_start_us":817,"threads_started":1}
I20260620 03:00:17.168296 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:17.155386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36952 (local address 127.0.97.1:44593)
0620 03:00:17.157389 (+ 2003us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:17.157398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:17.159066 (+ 1668us) server_negotiation.cc:407] Connection header received
0620 03:00:17.159421 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:17.159434 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:17.159776 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:17.159991 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:00:17.161534 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.162978 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.165449 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.166885 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.167028 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:17.167258 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:17.167448 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 03:00:17.167597 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1825,"thread_start_us":2547,"threads_started":1}
I20260620 03:00:17.168857 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:17.154707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37416 (local address 127.0.97.2:37675)
0620 03:00:17.159868 (+ 5161us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:17.159874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:17.159896 (+ 22us) server_negotiation.cc:407] Connection header received
0620 03:00:17.159969 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:17.159974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:17.160135 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:17.160334 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:00:17.161411 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.162979 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.164918 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.166436 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.167957 (+ 1521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:17.168178 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:17.168440 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 03:00:17.168584 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5014,"thread_start_us":4899,"threads_started":1}
I20260620 03:00:17.169620 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:17.157137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34579 (local address 127.0.0.1:40184)
0620 03:00:17.161268 (+ 4131us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:17.161290 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:17.161459 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:17.162330 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:17.162336 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:17.162360 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:00:17.162699 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:17.162708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.164663 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.164668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:17.166588 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:17.166602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.168869 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.168878 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:17.169058 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:17.169094 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 03:00:17.169186 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 03:00:17.169316 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3972,"thread_start_us":3551,"threads_started":1}
I20260620 03:00:17.169922 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:17.157707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.0.97.3:34579)
0620 03:00:17.159442 (+ 1735us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:17.159454 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:17.161369 (+ 1915us) server_negotiation.cc:407] Connection header received
0620 03:00:17.161676 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:17.161685 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:17.161902 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:17.162049 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:00:17.162878 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.164515 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.166763 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.168668 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.168791 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:17.169333 (+ 542us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:17.169406 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 03:00:17.169597 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1523,"thread_start_us":1323,"threads_started":1}
I20260620 03:00:17.172240 10844 tablet_service.cc:1467] Processing CreateTablet for tablet c9500495dec54363bc49e2c1a7c0128b (DEFAULT_TABLE table=test-table [id=f8da0024fdec4f59bf48970567424557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:00:17.172605 10978 tablet_service.cc:1467] Processing CreateTablet for tablet c9500495dec54363bc49e2c1a7c0128b (DEFAULT_TABLE table=test-table [id=f8da0024fdec4f59bf48970567424557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:00:17.173554 10844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9500495dec54363bc49e2c1a7c0128b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:00:17.173959 10978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9500495dec54363bc49e2c1a7c0128b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:00:17.173888 11111 tablet_service.cc:1467] Processing CreateTablet for tablet c9500495dec54363bc49e2c1a7c0128b (DEFAULT_TABLE table=test-table [id=f8da0024fdec4f59bf48970567424557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:00:17.175181 11111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9500495dec54363bc49e2c1a7c0128b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:00:17.186319 11194 tablet_bootstrap.cc:492] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123: Bootstrap starting.
I20260620 03:00:17.188310 11195 tablet_bootstrap.cc:492] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6: Bootstrap starting.
I20260620 03:00:17.189527 11196 tablet_bootstrap.cc:492] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6: Bootstrap starting.
I20260620 03:00:17.190260 11194 tablet_bootstrap.cc:654] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123: Neither blocks nor log segments found. Creating new log.
I20260620 03:00:17.191532 11194 log.cc:824] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123: Log is configured to *not* fsync() on all Append() calls
I20260620 03:00:17.191701 11195 tablet_bootstrap.cc:654] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6: Neither blocks nor log segments found. Creating new log.
I20260620 03:00:17.192298 11196 tablet_bootstrap.cc:654] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6: Neither blocks nor log segments found. Creating new log.
I20260620 03:00:17.192811 11195 log.cc:824] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6: Log is configured to *not* fsync() on all Append() calls
I20260620 03:00:17.193455 11196 log.cc:824] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6: Log is configured to *not* fsync() on all Append() calls
I20260620 03:00:17.195760 11194 tablet_bootstrap.cc:492] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123: No bootstrap required, opened a new log
I20260620 03:00:17.196082 11194 ts_tablet_manager.cc:1397] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 03:00:17.200248 11195 tablet_bootstrap.cc:492] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6: No bootstrap required, opened a new log
I20260620 03:00:17.200515 11195 ts_tablet_manager.cc:1397] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20260620 03:00:17.203037 11196 tablet_bootstrap.cc:492] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6: No bootstrap required, opened a new log
I20260620 03:00:17.203313 11196 ts_tablet_manager.cc:1397] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20260620 03:00:17.206928 11194 raft_consensus.cc:348] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0fa42927ee437f82ae1da8c0ac94c6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37675 } } peers { permanent_uuid: "f17443026f2943059043ec77f4ce5123" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44593 } } peers { permanent_uuid: "d3f0dd85846b4c27801cf00f85954eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34579 } }
I20260620 03:00:17.207374 11194 raft_consensus.cc:374] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:00:17.207472 11194 raft_consensus.cc:729] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f17443026f2943059043ec77f4ce5123, State: Initialized, Role: FOLLOWER
I20260620 03:00:17.208128 11194 consensus_queue.cc:260] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0fa42927ee437f82ae1da8c0ac94c6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37675 } } peers { permanent_uuid: "f17443026f2943059043ec77f4ce5123" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44593 } } peers { permanent_uuid: "d3f0dd85846b4c27801cf00f85954eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34579 } }
I20260620 03:00:17.210810 11195 raft_consensus.cc:348] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0fa42927ee437f82ae1da8c0ac94c6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37675 } } peers { permanent_uuid: "f17443026f2943059043ec77f4ce5123" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44593 } } peers { permanent_uuid: "d3f0dd85846b4c27801cf00f85954eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34579 } }
I20260620 03:00:17.211237 11195 raft_consensus.cc:374] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:00:17.211337 11194 ts_tablet_manager.cc:1428] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260620 03:00:17.211333 11195 raft_consensus.cc:729] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f0fa42927ee437f82ae1da8c0ac94c6, State: Initialized, Role: FOLLOWER
I20260620 03:00:17.211508 11196 raft_consensus.cc:348] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0fa42927ee437f82ae1da8c0ac94c6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37675 } } peers { permanent_uuid: "f17443026f2943059043ec77f4ce5123" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44593 } } peers { permanent_uuid: "d3f0dd85846b4c27801cf00f85954eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34579 } }
I20260620 03:00:17.211877 11196 raft_consensus.cc:374] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:00:17.211961 11196 raft_consensus.cc:729] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f0dd85846b4c27801cf00f85954eb6, State: Initialized, Role: FOLLOWER
I20260620 03:00:17.211884 11195 consensus_queue.cc:260] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0fa42927ee437f82ae1da8c0ac94c6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37675 } } peers { permanent_uuid: "f17443026f2943059043ec77f4ce5123" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44593 } } peers { permanent_uuid: "d3f0dd85846b4c27801cf00f85954eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34579 } }
I20260620 03:00:17.212518 11196 consensus_queue.cc:260] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0fa42927ee437f82ae1da8c0ac94c6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37675 } } peers { permanent_uuid: "f17443026f2943059043ec77f4ce5123" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44593 } } peers { permanent_uuid: "d3f0dd85846b4c27801cf00f85954eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34579 } }
I20260620 03:00:17.213776 11176 heartbeater.cc:503] Master 127.0.97.62:33043 was elected leader, sending a full tablet report...
I20260620 03:00:17.214263 11196 ts_tablet_manager.cc:1428] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260620 03:00:17.218876 11043 heartbeater.cc:503] Master 127.0.97.62:33043 was elected leader, sending a full tablet report...
I20260620 03:00:17.219718 11195 ts_tablet_manager.cc:1428] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6: Time spent starting tablet: real 0.019s user 0.008s sys 0.012s
I20260620 03:00:17.223673 11200 raft_consensus.cc:482] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:00:17.223927 11200 raft_consensus.cc:504] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0fa42927ee437f82ae1da8c0ac94c6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37675 } } peers { permanent_uuid: "f17443026f2943059043ec77f4ce5123" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44593 } } peers { permanent_uuid: "d3f0dd85846b4c27801cf00f85954eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34579 } }
I20260620 03:00:17.225415 11200 leader_election.cc:288] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f0fa42927ee437f82ae1da8c0ac94c6 (127.0.97.2:37675), d3f0dd85846b4c27801cf00f85954eb6 (127.0.97.3:34579)
I20260620 03:00:17.232544 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:17.225682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60079 (local address 127.0.97.2:37675)
0620 03:00:17.225941 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:17.225948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:17.226608 (+ 660us) server_negotiation.cc:407] Connection header received
0620 03:00:17.226835 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:17.226838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:17.226892 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:17.226989 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:00:17.227592 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.228569 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.230592 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.231564 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.231665 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:17.231730 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:17.232143 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 03:00:17.232299 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 03:00:17.232587 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:17.225559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37675 (local address 127.0.97.1:60079)
0620 03:00:17.226489 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:17.226512 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:17.226673 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:17.227012 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:17.227017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:17.227053 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:00:17.227461 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:17.227474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.228685 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.228690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:17.230450 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:17.230461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.231764 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.231773 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:17.231945 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:17.231981 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 03:00:17.232081 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 03:00:17.232212 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":38,"thread_start_us":325,"threads_started":1}
I20260620 03:00:17.232949 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:17.226169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39325 (local address 127.0.97.3:34579)
0620 03:00:17.226366 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:17.226375 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:17.227386 (+ 1011us) server_negotiation.cc:407] Connection header received
0620 03:00:17.227505 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:17.227512 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:17.227593 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:17.227663 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:00:17.228264 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.229204 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.231122 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.232030 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.232142 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:17.232232 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:17.232585 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 03:00:17.232719 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 03:00:17.233088 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:17.225727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34579 (local address 127.0.97.1:39325)
0620 03:00:17.227287 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0620 03:00:17.227305 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:00:17.227396 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:00:17.227711 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:00:17.227716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:00:17.227739 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:00:17.228100 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:17.228110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.229336 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.229340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:17.230984 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:00:17.230997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.232233 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.232240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:00:17.232407 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:17.232442 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 03:00:17.232541 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 03:00:17.232672 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1426,"mutex_wait_us":17,"thread_start_us":1227,"threads_started":1}
I20260620 03:00:17.233522 10998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9500495dec54363bc49e2c1a7c0128b" candidate_uuid: "f17443026f2943059043ec77f4ce5123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f0fa42927ee437f82ae1da8c0ac94c6" is_pre_election: true
I20260620 03:00:17.234117 10998 raft_consensus.cc:2454] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f17443026f2943059043ec77f4ce5123 in term 0.
I20260620 03:00:17.234086 11131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9500495dec54363bc49e2c1a7c0128b" candidate_uuid: "f17443026f2943059043ec77f4ce5123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3f0dd85846b4c27801cf00f85954eb6" is_pre_election: true
I20260620 03:00:17.234632 11131 raft_consensus.cc:2454] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f17443026f2943059043ec77f4ce5123 in term 0.
I20260620 03:00:17.234758 10800 leader_election.cc:302] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f0fa42927ee437f82ae1da8c0ac94c6, f17443026f2943059043ec77f4ce5123; no voters:
I20260620 03:00:17.235167 11200 raft_consensus.cc:2781] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:00:17.235321 11200 raft_consensus.cc:482] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:00:17.235451 11200 raft_consensus.cc:3037] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:00:17.237743 11200 raft_consensus.cc:504] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0fa42927ee437f82ae1da8c0ac94c6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37675 } } peers { permanent_uuid: "f17443026f2943059043ec77f4ce5123" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44593 } } peers { permanent_uuid: "d3f0dd85846b4c27801cf00f85954eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34579 } }
I20260620 03:00:17.238415 11200 leader_election.cc:288] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [CANDIDATE]: Term 1 election: Requested vote from peers 9f0fa42927ee437f82ae1da8c0ac94c6 (127.0.97.2:37675), d3f0dd85846b4c27801cf00f85954eb6 (127.0.97.3:34579)
I20260620 03:00:17.238826 10998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9500495dec54363bc49e2c1a7c0128b" candidate_uuid: "f17443026f2943059043ec77f4ce5123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f0fa42927ee437f82ae1da8c0ac94c6"
I20260620 03:00:17.238931 11131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9500495dec54363bc49e2c1a7c0128b" candidate_uuid: "f17443026f2943059043ec77f4ce5123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3f0dd85846b4c27801cf00f85954eb6"
I20260620 03:00:17.239032 10998 raft_consensus.cc:3037] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:00:17.239116 11131 raft_consensus.cc:3037] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:00:17.242233 11131 raft_consensus.cc:2454] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f17443026f2943059043ec77f4ce5123 in term 1.
I20260620 03:00:17.242233 10998 raft_consensus.cc:2454] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f17443026f2943059043ec77f4ce5123 in term 1.
I20260620 03:00:17.242640 10800 leader_election.cc:302] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f0fa42927ee437f82ae1da8c0ac94c6, f17443026f2943059043ec77f4ce5123; no voters:
I20260620 03:00:17.242931 11200 raft_consensus.cc:2781] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:00:17.243368 11200 raft_consensus.cc:686] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [term 1 LEADER]: Becoming Leader. State: Replica: f17443026f2943059043ec77f4ce5123, State: Running, Role: LEADER
I20260620 03:00:17.243901 11200 consensus_queue.cc:237] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0fa42927ee437f82ae1da8c0ac94c6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37675 } } peers { permanent_uuid: "f17443026f2943059043ec77f4ce5123" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44593 } } peers { permanent_uuid: "d3f0dd85846b4c27801cf00f85954eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34579 } }
I20260620 03:00:17.248371 10719 catalog_manager.cc:5495] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 reported cstate change: term changed from 0 to 1, leader changed from <none> to f17443026f2943059043ec77f4ce5123 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "f17443026f2943059043ec77f4ce5123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0fa42927ee437f82ae1da8c0ac94c6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f17443026f2943059043ec77f4ce5123" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3f0dd85846b4c27801cf00f85954eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34579 } health_report { overall_health: UNKNOWN } } }
I20260620 03:00:17.264205 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:17.259513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36968 (local address 127.0.97.1:44593)
0620 03:00:17.259683 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:17.259688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:17.259772 (+ 84us) server_negotiation.cc:407] Connection header received
0620 03:00:17.259935 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:17.259939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:17.260010 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:17.260130 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:00:17.260440 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.261141 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.262413 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.262803 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.262897 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:17.263297 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:00:17.263774 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:00:17.263895 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:17.263952 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 03:00:17.264008 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 03:00:17.272058 11131 raft_consensus.cc:1261] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6 [term 1 FOLLOWER]: Refusing update from remote peer f17443026f2943059043ec77f4ce5123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:00:17.272058 10998 raft_consensus.cc:1261] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6 [term 1 FOLLOWER]: Refusing update from remote peer f17443026f2943059043ec77f4ce5123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:00:17.273094 11205 consensus_queue.cc:1035] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f0fa42927ee437f82ae1da8c0ac94c6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:00:17.273452 11200 consensus_queue.cc:1035] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3f0dd85846b4c27801cf00f85954eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:00:17.287935 11209 mvcc.cc:205] Tried to move back new op lower bound from 7298762413135695872 to 7298762413034704896. Current Snapshot: MvccSnapshot[applied={T|T < 7298762413135695872}]
I20260620 03:00:17.304693 10719 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:00:17.304983 10719 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:00:17.308194 10719 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc609528baf04c9bb702d600058ba37d: Sending DeleteTablet for 3 replicas of tablet c9500495dec54363bc49e2c1a7c0128b
I20260620 03:00:17.309211 10978 tablet_service.cc:1514] Processing DeleteTablet for tablet c9500495dec54363bc49e2c1a7c0128b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:00:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37416
I20260620 03:00:17.309948 11111 tablet_service.cc:1514] Processing DeleteTablet for tablet c9500495dec54363bc49e2c1a7c0128b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:00:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40184
I20260620 03:00:17.310554 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 10781
I20260620 03:00:17.313256 10844 tablet_service.cc:1514] Processing DeleteTablet for tablet c9500495dec54363bc49e2c1a7c0128b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:00:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36952
I20260620 03:00:17.328619 11220 tablet_replica.cc:330] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123: stopping tablet replica
I20260620 03:00:17.329071 11220 raft_consensus.cc:2229] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:00:17.329768 11220 raft_consensus.cc:2258] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:00:17.331784 11220 ts_tablet_manager.cc:1905] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:00:17.334998 11218 tablet_replica.cc:330] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6: stopping tablet replica
I20260620 03:00:17.335429 11218 raft_consensus.cc:2229] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:00:17.335932 11218 raft_consensus.cc:2258] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 03:00:17 dist-test-slave-f1j0 krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:00:17.336692 11217 tablet_replica.cc:330] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6: stopping tablet replica
I20260620 03:00:17.337080 11217 raft_consensus.cc:2229] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:00:17.337561 11217 raft_consensus.cc:2258] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:00:17.339175 11220 ts_tablet_manager.cc:1918] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:00:17.339354 11220 log.cc:1196] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/c9500495dec54363bc49e2c1a7c0128b
I20260620 03:00:17.339411 11218 ts_tablet_manager.cc:1905] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:00:17.339949 11220 ts_tablet_manager.cc:1939] T c9500495dec54363bc49e2c1a7c0128b P f17443026f2943059043ec77f4ce5123: Deleting consensus metadata
I20260620 03:00:17.341758 10705 catalog_manager.cc:4841] TS f17443026f2943059043ec77f4ce5123 (127.0.97.1:44593): tablet c9500495dec54363bc49e2c1a7c0128b (table test-table [id=f8da0024fdec4f59bf48970567424557]) successfully deleted
I20260620 03:00:17.345933 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:17.311930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36980 (local address 127.0.97.1:44593)
0620 03:00:17.312131 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:17.312140 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:17.324223 (+ 12083us) server_negotiation.cc:407] Connection header received
0620 03:00:17.324687 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:17.324696 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:17.324809 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:17.324891 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:17.326932 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.328400 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.330990 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.331898 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.332011 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:17.338245 (+ 6234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:17.338257 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:17.338274 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:00:17.338328 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:17.342440 (+ 4112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:00:17.343264 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:00:17.343275 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:00:17.343281 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:00:17.343460 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:00:17.343932 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:00:17.343937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:00:17.343941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:00:17.344346 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:17.344670 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:17.345241 (+ 571us) server_negotiation.cc:299] Negotiation successful
0620 03:00:17.345482 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 03:00:17.347088 10904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:00:17.347170 11217 ts_tablet_manager.cc:1905] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:00:17.351862 11218 ts_tablet_manager.cc:1918] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:00:17.352036 11218 log.cc:1196] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/c9500495dec54363bc49e2c1a7c0128b
I20260620 03:00:17.352473 11218 ts_tablet_manager.cc:1939] T c9500495dec54363bc49e2c1a7c0128b P d3f0dd85846b4c27801cf00f85954eb6: Deleting consensus metadata
I20260620 03:00:17.354033 10705 catalog_manager.cc:4841] TS d3f0dd85846b4c27801cf00f85954eb6 (127.0.97.3:34579): tablet c9500495dec54363bc49e2c1a7c0128b (table test-table [id=f8da0024fdec4f59bf48970567424557]) successfully deleted
I20260620 03:00:17.354552 11217 ts_tablet_manager.cc:1918] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:00:17.354729 11217 log.cc:1196] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/c9500495dec54363bc49e2c1a7c0128b
I20260620 03:00:17.355221 11217 ts_tablet_manager.cc:1939] T c9500495dec54363bc49e2c1a7c0128b P 9f0fa42927ee437f82ae1da8c0ac94c6: Deleting consensus metadata
I20260620 03:00:17.356931 10707 catalog_manager.cc:4841] TS 9f0fa42927ee437f82ae1da8c0ac94c6 (127.0.97.2:37675): tablet c9500495dec54363bc49e2c1a7c0128b (table test-table [id=f8da0024fdec4f59bf48970567424557]) successfully deleted
I20260620 03:00:17.446415 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 10781
I20260620 03:00:17.469760 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 10915
Jun 20 03:00:17 dist-test-slave-f1j0 krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:00:17.485285 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:17.470554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.0.97.2:37675)
0620 03:00:17.470731 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:17.470735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:17.470752 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:00:17.470806 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:17.470810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:17.470873 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:17.470955 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:17.472247 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.473090 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.474341 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.474718 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.474817 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:17.479043 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:17.479058 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:17.479082 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:00:17.479120 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:17.482473 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:00:17.483197 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:00:17.483205 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:00:17.483208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:00:17.483331 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:00:17.483700 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:00:17.483706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:00:17.483756 (+ 50us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:00:17.484144 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:17.484433 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:17.484728 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 03:00:17.484939 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 03:00:17.486002 11038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:00:17.574146 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 10915
I20260620 03:00:17.591948 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 11048
Jun 20 03:00:17 dist-test-slave-f1j0 krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:00:17.606055 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:00:17.592703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.0.97.3:34579)
0620 03:00:17.592893 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 03:00:17.592898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:00:17.592916 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:00:17.593036 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:00:17.593040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:00:17.593101 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:00:17.593191 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:00:17.594453 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.595155 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.596360 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:00:17.596728 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:00:17.596814 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:00:17.600872 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:00:17.600887 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:00:17.600904 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:00:17.600935 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:00:17.603525 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:00:17.604118 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:00:17.604125 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:00:17.604127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:00:17.604276 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:00:17.604651 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:00:17.604654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:00:17.604656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:00:17.604977 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:00:17.605179 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 03:00:17.605454 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 03:00:17.605686 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 03:00:17.606696 11171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:00:17.696012 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 11048
I20260620 03:00:17.713485 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 10688
I20260620 03:00:17.714614 10750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:00:17.823999 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 10688
2026-06-20T03:00:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4119 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:00:17 dist-test-slave-f1j0 krb5kdc[11227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:00:17 dist-test-slave-f1j0 krb5kdc[11227](info): set up 2 sockets
Jun 20 03:00:17 dist-test-slave-f1j0 krb5kdc[11227](info): commencing operation
krb5kdc: starting...
W20260620 03:00:18.926513 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:00:19 dist-test-slave-f1j0 krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:00:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:00:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 03:00:19.257773 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpRaC98/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/postgres -l logfile start
2026-06-20 03:00:21.735 UTC [11257] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 03:00:21.735 UTC [11257] LOG: listening on IPv4 address "127.0.97.20", port 35051
2026-06-20 03:00:21.739 UTC [11257] LOG: listening on Unix socket "/tmp/.s.PGSQL.35051"
2026-06-20 03:00:21.762 UTC [11260] LOG: database system was shut down at 2026-06-20 03:00:20 UTC
2026-06-20 03:00:21.765 UTC [11257] LOG: database system is ready to accept connections
I20260620 03:00:22.754139 388 mini_postgres.cc:96] Postgres bound to 35051
2026-06-20 03:00:22.771 UTC [11268] FATAL: database "slave" does not exist
127.0.97.20:35051 - accepting connections
I20260620 03:00:22.772464 388 mini_ranger.cc:161] Starting Ranger...
I20260620 03:00:22.798712 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 03:00:23.284582 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 03:00:23.284745 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ranger-admin
2026-06-20 03:00:23,591 [I] DB FLAVOR :POSTGRES
2026-06-20 03:00:23,592 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 03:00:23,592 [I] Checking connection..
2026-06-20 03:00:23,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 03:00:23,833 [I] Checking connection passed.
2026-06-20 03:00:23,833 [I] --------- Verifying version history table ---------
2026-06-20 03:00:23,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 03:00:24,070 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 03:00:24,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 03:00:24,306 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 03:00:24,306 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 03:00:24,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 03:00:24.571 UTC [11332] WARNING: there is no transaction in progress
2026-06-20 03:00:24,577 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 03:00:24,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 03:00:24,839 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 03:00:24,839 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 03:00:24,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 03:00:25,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 03:00:25,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f1j0.c.gcp-upstream.internal','N') ;"
2026-06-20 03:00:25,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 03:00:25,830 [I] Table x_portal_user does not exist in database ranger
2026-06-20 03:00:25,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 03:00:26,058 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 03:00:26,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 03:00:26,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 03:00:26,567 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 03:00:26,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 03:00:27.515 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 03:00:27.532 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 03:00:27.548 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 03:00:27.562 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 03:00:27.580 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 03:00:27.600 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 03:00:27.665 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 03:00:27.672 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 03:00:27.686 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 03:00:27.697 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 03:00:27.707 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 03:00:27.721 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 03:00:27.730 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 03:00:27.736 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 03:00:27.743 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 03:00:27.750 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 03:00:28,360 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 03:00:28,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 03:00:28,581 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 03:00:28,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 03:00:28,822 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 03:00:28,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 03:00:29,073 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 03:00:29,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 03:00:29,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 03:00:29,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f1j0.c.gcp-upstream.internal';"
2026-06-20 03:00:29,807 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 03:00:29,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 03:00:30,076 [I] DB_PATCHES have already been applied
I20260620 03:00:30.082093 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpRaC98/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 03:00:30.082239 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 03:00:30.100730 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:47767
Jun 20, 2026 3:00:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:00:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:00:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 3:00:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 3:00:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 3:00:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 3:00:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 03:00:31.232926 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.460s user 0.003s sys 0.039s
Jun 20, 2026 3:00:51 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 3:00:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 3:00:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 3:00:51 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 3:00:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 3:00:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 3:00:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 3:00:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 3:00:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 3:00:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 3:00:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 3:00:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 3:00:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 3:00:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 3:00:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 3:00:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 3:00:51 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 3:00:52 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 03:00:52.857266 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 03:00:53 dist-test-slave-f1j0 krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924453, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 03:00:53.022529 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 03:00:53.047227 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 03:00:53.904961 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 03:00:53.905105 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ranger-kms
I20260620 03:00:53.905261 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:35051
2026-06-20 03:00:54,319 [I] DB FLAVOR :POSTGRES
2026-06-20 03:00:54,319 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 03:00:54,319 [I] Checking connection
2026-06-20 03:00:54,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 03:00:54,564 [I] connection success
2026-06-20 03:00:54,565 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 03:00:54,565 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 03:00:54,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 03:00:54,794 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 03:00:54,795 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 03:00:54,795 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 03:00:54,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35051/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 03:00:55,043 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 03:00:55.509506 388 mini_ranger_kms.cc:310] Created kms service
I20260620 03:00:55.683569 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 03:00:55.727144 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 03:00:55 dist-test-slave-f1j0 krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924453, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:00:56.117252 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 03:00:56.117590 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 03:00:56.117648 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 03:00:56.139356 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11620
I20260620 03:00:56.139562 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:59719
Jun 20, 2026 3:00:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:00:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:00:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 3:00:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 3:00:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 3:00:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 03:00:58 dist-test-slave-f1j0 krb5kdc[11227](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781924458, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:00:58.462287 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.440s user 0.123s sys 0.320s
I20260620 03:00:58.462555 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 03:00:58.462767 388 mini_ranger_kms.cc:401] 127.0.97.20:59719/kms/v1/keys
Jun 20 03:00:58 dist-test-slave-f1j0 krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924458, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 03:00:59 dist-test-slave-f1j0 krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924458, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 03:01:01 dist-test-slave-f1j0 krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:01:01.466885 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37509
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37509
--ranger_config_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59719/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:01:01.817883 11671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:01:01.818183 11671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:01:01.818288 11671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:01:01.825510 11671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:01:01.825678 11671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:01:01.825742 11671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:01:01.825804 11671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:01:01.825852 11671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:01:01.835947 11671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:59719/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37509
--ranger_config_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:01:01.837177 11671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:01:01.838538 11671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:01:01.850857 11676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:01.850791 11679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:01.850759 11677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:01:01.851718 11671 server_base.cc:1034] running on GCE node
I20260620 03:01:01.852519 11671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:01:01.854003 11671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:01:01.855197 11671 hybrid_clock.cc:648] HybridClock initialized: now 1781924461855152 us; error 48 us; skew 500 ppm
I20260620 03:01:01.855508 11671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:01:01 dist-test-slave-f1j0 krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:01:01.858546 11671 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:01:01.859938 11671 webserver.cc:460] Webserver started at http://127.0.97.62:44535/ using document root <none> and password file <none>
I20260620 03:01:01.860590 11671 fs_manager.cc:359] Metadata directory not provided
I20260620 03:01:01.860694 11671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:01:01.861023 11671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 03:01:01 dist-test-slave-f1j0 krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:01:01.919641 11671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "c7377f6626244abba729f49807f4711e"
format_stamp: "Formatted at 2026-06-20 03:01:01 on dist-test-slave-f1j0"
server_key: "4f5b20d61daf2ac552a48ec12addd293"
server_key_iv: "45698171d9fee9ed4973375f9d6e42f9"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:01.920490 11671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "c7377f6626244abba729f49807f4711e"
format_stamp: "Formatted at 2026-06-20 03:01:01 on dist-test-slave-f1j0"
server_key: "4f5b20d61daf2ac552a48ec12addd293"
server_key_iv: "45698171d9fee9ed4973375f9d6e42f9"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:01.926435 11671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 03:01:01.946550 11686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:01.948202 11671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:01:01.948387 11671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "c7377f6626244abba729f49807f4711e"
format_stamp: "Formatted at 2026-06-20 03:01:01 on dist-test-slave-f1j0"
server_key: "4f5b20d61daf2ac552a48ec12addd293"
server_key_iv: "45698171d9fee9ed4973375f9d6e42f9"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:01.948601 11671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:01:01.971165 11671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:01:01.975416 11671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:01:01.975734 11671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:01:01.995615 11671 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37509
I20260620 03:01:01.995636 11737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37509 every 8 connection(s)
I20260620 03:01:01.997004 11671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:01:02.000821 11738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:01:02.005834 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 11671
I20260620 03:01:02.006028 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:01:02.006667 388 external_mini_cluster.cc:1428] Setting key 65710afc378500ef788ea4eb00f7f8b9
I20260620 03:01:02.013667 11738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e: Bootstrap starting.
I20260620 03:01:02.017370 11738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e: Neither blocks nor log segments found. Creating new log.
I20260620 03:01:02.018455 11738 log.cc:824] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e: Log is configured to *not* fsync() on all Append() calls
I20260620 03:01:02.021349 11738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e: No bootstrap required, opened a new log
Jun 20 03:01:02 dist-test-slave-f1j0 krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:01:02.030905 11741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:02.008787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58886 (local address 127.0.97.62:37509)
0620 03:01:02.009712 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:02.009728 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:02.009783 (+ 55us) server_negotiation.cc:407] Connection header received
0620 03:01:02.012216 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:02.012246 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:02.012771 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:02.013222 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:02.014504 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:02.015806 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:02.017304 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:02.018138 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:02.018355 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:02.022763 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:02.022771 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:02.022786 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:02.022822 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:02.026068 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:02.026919 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:02.026925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:02.026932 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:02.027072 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:02.027483 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:02.027486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:02.027488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:02.028045 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:02.028292 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:02.028735 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 03:01:02.029128 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":393,"threads_started":1}
I20260620 03:01:02.031257 11738 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7377f6626244abba729f49807f4711e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37509 } }
I20260620 03:01:02.031644 11738 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:01:02.031755 11738 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7377f6626244abba729f49807f4711e, State: Initialized, Role: FOLLOWER
I20260620 03:01:02.032356 11738 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7377f6626244abba729f49807f4711e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37509 } }
I20260620 03:01:02.032622 11738 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:01:02.032779 11738 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:01:02.032974 11738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:01:02.035269 11738 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7377f6626244abba729f49807f4711e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37509 } }
I20260620 03:01:02.035815 11738 leader_election.cc:302] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7377f6626244abba729f49807f4711e; no voters:
I20260620 03:01:02.036375 11738 leader_election.cc:288] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:01:02.036520 11743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:01:02.037199 11743 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e [term 1 LEADER]: Becoming Leader. State: Replica: c7377f6626244abba729f49807f4711e, State: Running, Role: LEADER
I20260620 03:01:02.037834 11743 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7377f6626244abba729f49807f4711e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37509 } }
I20260620 03:01:02.038515 11738 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:01:02.040724 11744 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7377f6626244abba729f49807f4711e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7377f6626244abba729f49807f4711e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37509 } } }
I20260620 03:01:02.041109 11744 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e [sys.catalog]: This master's current role is: LEADER
I20260620 03:01:02.040866 11745 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7377f6626244abba729f49807f4711e. Latest consensus state: current_term: 1 leader_uuid: "c7377f6626244abba729f49807f4711e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7377f6626244abba729f49807f4711e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37509 } } }
I20260620 03:01:02.041272 11745 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e [sys.catalog]: This master's current role is: LEADER
I20260620 03:01:02.043867 11748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:01:02.049296 11748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:01:02.050033 11738 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 03:01:02.074471 11748 catalog_manager.cc:1347] Generated new cluster ID: 5d03704f4eb94548994b6eb7638954b7
I20260620 03:01:02.074615 11748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:01:02.084353 11748 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:01:02.086310 11748 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:01:02.096710 11748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e: Generated new TSK 0
I20260620 03:01:02.097852 11748 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:01:02.847019 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59719/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37509
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:01:03 dist-test-slave-f1j0 krb5kdc[11227](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781924463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 03:01:03.269682 11783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:01:03.270185 11783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:01:03.270709 11783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:01:03.281803 11783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:01:03.281967 11783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:01:03.282143 11783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:01:03.298169 11783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:59719/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:01:03.300127 11783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:01:03.301931 11783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:01:03.320209 11794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:03.319770 11793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:01:03.328658 11783 server_base.cc:1034] running on GCE node
W20260620 03:01:03.329005 11797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:01:03.329888 11783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:01:03.331246 11783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:01:03.332567 11783 hybrid_clock.cc:648] HybridClock initialized: now 1781924463332504 us; error 48 us; skew 500 ppm
I20260620 03:01:03.332966 11783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:01:03 dist-test-slave-f1j0 krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:01:03.337558 11783 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:01:03.339970 11783 webserver.cc:460] Webserver started at http://127.0.97.1:38057/ using document root <none> and password file <none>
I20260620 03:01:03.340874 11783 fs_manager.cc:359] Metadata directory not provided
I20260620 03:01:03.341028 11783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:01:03.341377 11783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 03:01:03 dist-test-slave-f1j0 krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:01:03.364212 11783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "4a3ddebe2b674820a3e77a86e4424b33"
format_stamp: "Formatted at 2026-06-20 03:01:03 on dist-test-slave-f1j0"
server_key: "abaa94fd5814e6b51a10b02159b258e4"
server_key_iv: "b4d48dfdca25b1f786121cfb9dc4a088"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:03.365429 11783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4a3ddebe2b674820a3e77a86e4424b33"
format_stamp: "Formatted at 2026-06-20 03:01:03 on dist-test-slave-f1j0"
server_key: "abaa94fd5814e6b51a10b02159b258e4"
server_key_iv: "b4d48dfdca25b1f786121cfb9dc4a088"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:03.379957 11783 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.010s sys 0.001s
I20260620 03:01:03.402242 11803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:03.404774 11783 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260620 03:01:03.404990 11783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "4a3ddebe2b674820a3e77a86e4424b33"
format_stamp: "Formatted at 2026-06-20 03:01:03 on dist-test-slave-f1j0"
server_key: "abaa94fd5814e6b51a10b02159b258e4"
server_key_iv: "b4d48dfdca25b1f786121cfb9dc4a088"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:03.405264 11783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:01:03.457813 11783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:01:03.463896 11783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:01:03.464394 11783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:01:03.467903 11783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:01:03.469796 11783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:01:03.469902 11783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:03.470067 11783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:01:03.470141 11783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:03.536266 11783 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40683
I20260620 03:01:03.536288 11915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40683 every 8 connection(s)
I20260620 03:01:03.537794 11783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:01:03.542826 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 11783
I20260620 03:01:03.543066 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:01:03.544040 388 external_mini_cluster.cc:1428] Setting key 8180bed7723ecc9f303a9a0b739872ce
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Jun 20 03:01:03 dist-test-slave-f1j0 krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 03:01:03.586470 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:03.541873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37509 (local address 127.0.97.1:39423)
0620 03:01:03.554536 (+ 12663us) negotiation.cc:107] Waiting for socket to connect
0620 03:01:03.554592 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 03:01:03.557327 (+ 2735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:01:03.563106 (+ 5779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:01:03.563123 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:01:03.563849 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:01:03.564738 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:03.564848 (+ 110us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:03.566635 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:03.566643 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:03.568932 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:03.568953 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:03.570061 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:03.570071 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:03.570442 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:03.572425 (+ 1983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:01:03.572461 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:01:03.575141 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:01:03.578489 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:01:03.578500 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:01:03.578521 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:01:03.581904 (+ 3383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:01:03.582384 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:01:03.582392 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:01:03.582399 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:01:03.582700 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:01:03.583496 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:01:03.583507 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:01:03.583996 (+ 489us) client_negotiation.cc:769] Sending connection context
0620 03:01:03.584306 (+ 310us) client_negotiation.cc:240] Negotiation successful
0620 03:01:03.584910 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12284,"thread_start_us":15573,"threads_started":1}
I20260620 03:01:03.587736 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:03.541009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39423 (local address 127.0.97.62:37509)
0620 03:01:03.562019 (+ 21010us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:03.562028 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:03.562055 (+ 27us) server_negotiation.cc:407] Connection header received
0620 03:01:03.562362 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:03.562370 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:03.562481 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:03.562586 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:03.565037 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:03.566429 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:03.569104 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:03.569537 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:03.569727 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:03.575420 (+ 5693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:03.575425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:03.575428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:03.575461 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:03.578337 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:03.582066 (+ 3729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:03.582075 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:03.582082 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:03.582239 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:03.582861 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:03.582869 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:03.582875 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:03.583262 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:03.583406 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:03.586960 (+ 3554us) server_negotiation.cc:299] Negotiation successful
0620 03:01:03.587228 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20859,"thread_start_us":20746,"threads_started":1}
I20260620 03:01:03.589993 11916 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37509
I20260620 03:01:03.590344 11916 heartbeater.cc:464] Registering TS with master...
I20260620 03:01:03.591166 11916 heartbeater.cc:511] Master 127.0.97.62:37509 requested a full tablet report, sending...
I20260620 03:01:03.593675 11703 ts_manager.cc:194] Registered new tserver with Master: 4a3ddebe2b674820a3e77a86e4424b33 (127.0.97.1:40683)
I20260620 03:01:03.596320 11703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:01:03.613674 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59719/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37509
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:01:03 dist-test-slave-f1j0 krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 03:01:04 dist-test-slave-f1j0 krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 03:01:04.136581 11922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:01:04.137104 11922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:01:04.137295 11922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:01:04.152410 11922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:01:04.152617 11922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:01:04.152835 11922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:01:04.167879 11922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:59719/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:01:04.169876 11922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:01:04.171604 11922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:01:04.219709 11929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:04.224912 11930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:01:04.229907 11922 server_base.cc:1034] running on GCE node
W20260620 03:01:04.238382 11933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:01:04.239557 11922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:01:04.240996 11922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:01:04.242256 11922 hybrid_clock.cc:648] HybridClock initialized: now 1781924464242200 us; error 50 us; skew 500 ppm
I20260620 03:01:04.242667 11922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:01:04 dist-test-slave-f1j0 krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:01:04.246829 11922 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:01:04.286105 11922 webserver.cc:460] Webserver started at http://127.0.97.2:37327/ using document root <none> and password file <none>
I20260620 03:01:04.287281 11922 fs_manager.cc:359] Metadata directory not provided
I20260620 03:01:04.287513 11922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:01:04.287926 11922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 03:01:04 dist-test-slave-f1j0 krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:01:04.307716 11922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "085958e9f7404fbe880f3c178afaddd0"
format_stamp: "Formatted at 2026-06-20 03:01:04 on dist-test-slave-f1j0"
server_key: "13ee45fbaee7e92c0a76d583cbec3ce9"
server_key_iv: "24da2db83f1613e31d8c5c3caa785087"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:04.308954 11922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "085958e9f7404fbe880f3c178afaddd0"
format_stamp: "Formatted at 2026-06-20 03:01:04 on dist-test-slave-f1j0"
server_key: "13ee45fbaee7e92c0a76d583cbec3ce9"
server_key_iv: "24da2db83f1613e31d8c5c3caa785087"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:04.323401 11922 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.008s sys 0.004s
I20260620 03:01:04.370242 11939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:04.372347 11922 fs_manager.cc:725] Time spent opening block manager: real 0.024s user 0.010s sys 0.011s
I20260620 03:01:04.372610 11922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "085958e9f7404fbe880f3c178afaddd0"
format_stamp: "Formatted at 2026-06-20 03:01:04 on dist-test-slave-f1j0"
server_key: "13ee45fbaee7e92c0a76d583cbec3ce9"
server_key_iv: "24da2db83f1613e31d8c5c3caa785087"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:04.373046 11922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:01:04.435441 11922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:01:04.441159 11922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:01:04.441591 11922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:01:04.443387 11922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:01:04.445423 11922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:01:04.445586 11922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:04.445770 11922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:01:04.445847 11922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:04.488390 11922 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44099
I20260620 03:01:04.488392 12058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44099 every 8 connection(s)
I20260620 03:01:04.490345 11922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:01:04.495102 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 11922
I20260620 03:01:04.495285 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:01:04.495820 388 external_mini_cluster.cc:1428] Setting key 39c46fd184cdc306205cffa9e1c616c3
Jun 20 03:01:04 dist-test-slave-f1j0 krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:01:04.519213 12061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:04.493368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45289 (local address 127.0.97.62:37509)
0620 03:01:04.493900 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:04.493907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:04.496503 (+ 2596us) server_negotiation.cc:407] Connection header received
0620 03:01:04.496976 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:04.496981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:04.497093 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:04.497208 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:04.501717 (+ 4509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:04.502984 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:04.505693 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:04.506459 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:04.506618 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:04.511428 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:04.511436 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:04.511441 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:04.511482 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:04.515159 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:04.516250 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:04.516256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:04.516260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:04.516416 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:04.517052 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:04.517057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:04.517060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:04.517434 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:04.517605 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:04.518573 (+ 968us) server_negotiation.cc:299] Negotiation successful
0620 03:01:04.518831 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":272,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 03:01:04.520449 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:04.493945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37509 (local address 127.0.97.2:45289)
0620 03:01:04.494789 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0620 03:01:04.494841 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 03:01:04.496685 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:01:04.497471 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:01:04.497480 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:01:04.498105 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:01:04.498967 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:04.498994 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:04.503204 (+ 4210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:04.503211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:04.505452 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:04.505480 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:04.506726 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:04.506733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:04.507508 (+ 775us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:04.508378 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:01:04.508410 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:01:04.511136 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:01:04.515324 (+ 4188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:01:04.515336 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:01:04.515360 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:01:04.516069 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:01:04.516549 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:01:04.516557 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:01:04.516564 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:01:04.516883 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:01:04.517607 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:01:04.517616 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:01:04.518111 (+ 495us) client_negotiation.cc:769] Sending connection context
0620 03:01:04.518442 (+ 331us) client_negotiation.cc:240] Negotiation successful
0620 03:01:04.519048 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":349,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 03:01:04.522202 12059 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37509
I20260620 03:01:04.522532 12059 heartbeater.cc:464] Registering TS with master...
I20260620 03:01:04.523243 12059 heartbeater.cc:511] Master 127.0.97.62:37509 requested a full tablet report, sending...
I20260620 03:01:04.524703 11703 ts_manager.cc:194] Registered new tserver with Master: 085958e9f7404fbe880f3c178afaddd0 (127.0.97.2:44099)
I20260620 03:01:04.525681 11703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:01:04.553555 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59719/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37509
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:01:04.600483 11916 heartbeater.cc:503] Master 127.0.97.62:37509 was elected leader, sending a full tablet report...
W20260620 03:01:04.895996 12065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:01:04.896379 12065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:01:04.896562 12065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:01:04.904173 12065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:01:04.904327 12065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:01:04.904510 12065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:01:04.914325 12065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:59719/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:01:04.915606 12065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:01:04.916972 12065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:01:04.930220 12070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:04.930200 12071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:04.930586 12073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:01:04.930954 12065 server_base.cc:1034] running on GCE node
I20260620 03:01:04.931543 12065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:01:04.932369 12065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:01:04.933609 12065 hybrid_clock.cc:648] HybridClock initialized: now 1781924464933512 us; error 50 us; skew 500 ppm
I20260620 03:01:04.933938 12065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:01:04 dist-test-slave-f1j0 krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:01:04.937534 12065 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:01:04.939044 12065 webserver.cc:460] Webserver started at http://127.0.97.3:43051/ using document root <none> and password file <none>
I20260620 03:01:04.939741 12065 fs_manager.cc:359] Metadata directory not provided
I20260620 03:01:04.939870 12065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:01:04.940189 12065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 03:01:04 dist-test-slave-f1j0 krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:01:04.955972 12065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "bec41d857fd44a48b3aeb9b2871d382e"
format_stamp: "Formatted at 2026-06-20 03:01:04 on dist-test-slave-f1j0"
server_key: "7ffa41b1fd67400b2261ab6d600ad4ef"
server_key_iv: "7f0af82b8b88151f6e3b2ff9af3c6bbe"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:04.956816 12065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bec41d857fd44a48b3aeb9b2871d382e"
format_stamp: "Formatted at 2026-06-20 03:01:04 on dist-test-slave-f1j0"
server_key: "7ffa41b1fd67400b2261ab6d600ad4ef"
server_key_iv: "7f0af82b8b88151f6e3b2ff9af3c6bbe"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:04.962258 12065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 03:01:04.978273 12080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:04.980062 12065 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 03:01:04.980219 12065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "bec41d857fd44a48b3aeb9b2871d382e"
format_stamp: "Formatted at 2026-06-20 03:01:04 on dist-test-slave-f1j0"
server_key: "7ffa41b1fd67400b2261ab6d600ad4ef"
server_key_iv: "7f0af82b8b88151f6e3b2ff9af3c6bbe"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:04.980465 12065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:01:04.997493 12065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:01:05.002851 12065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:01:05.003190 12065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:01:05.004364 12065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:01:05.006141 12065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:01:05.006237 12065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:05.006358 12065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:01:05.006422 12065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:05.041494 12065 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46049
I20260620 03:01:05.041513 12192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46049 every 8 connection(s)
I20260620 03:01:05.042786 12065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:01:05.048885 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 12065
I20260620 03:01:05.049079 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:01:05.049557 388 external_mini_cluster.cc:1428] Setting key 55d06b9bd74d6a21084b81474a20fec5
Jun 20 03:01:05 dist-test-slave-f1j0 krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:01:05.062777 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:05.045203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36683 (local address 127.0.97.62:37509)
0620 03:01:05.045768 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:05.045772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:05.046590 (+ 818us) server_negotiation.cc:407] Connection header received
0620 03:01:05.048314 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:05.048319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:05.048428 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:05.048545 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:05.050154 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.051183 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.053306 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.053822 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.053931 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:05.057552 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:05.057557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:05.057560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:05.057638 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:05.060006 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:05.060762 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:05.060766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:05.060768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:05.060889 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:05.061231 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:05.061234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:05.061236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:05.061487 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:05.061658 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:05.062364 (+ 706us) server_negotiation.cc:299] Negotiation successful
0620 03:01:05.062529 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":283,"threads_started":1}
I20260620 03:01:05.064041 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:05.045744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37509 (local address 127.0.97.3:36683)
0620 03:01:05.046432 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0620 03:01:05.046475 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 03:01:05.048064 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:01:05.048753 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:01:05.048761 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:01:05.049300 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:01:05.049962 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:05.049980 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.051405 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.051412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:05.053159 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:05.053170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.054002 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.054007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:05.054198 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:05.054871 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:01:05.054893 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:01:05.057352 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:01:05.060183 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:01:05.060190 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:01:05.060203 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:01:05.060658 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:01:05.060979 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:01:05.060982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:01:05.060985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:01:05.061142 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:01:05.061745 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:01:05.061751 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:01:05.062048 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 03:01:05.062284 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 03:01:05.062670 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":292,"threads_started":1}
I20260620 03:01:05.065419 12193 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37509
I20260620 03:01:05.065759 12193 heartbeater.cc:464] Registering TS with master...
I20260620 03:01:05.066471 12193 heartbeater.cc:511] Master 127.0.97.62:37509 requested a full tablet report, sending...
I20260620 03:01:05.067826 11702 ts_manager.cc:194] Registered new tserver with Master: bec41d857fd44a48b3aeb9b2871d382e (127.0.97.3:46049)
I20260620 03:01:05.068713 11702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36683
I20260620 03:01:05.077035 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:01:05.093829 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:05.081324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44322 (local address 127.0.97.62:37509)
0620 03:01:05.081483 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:05.081487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:05.082019 (+ 532us) server_negotiation.cc:407] Connection header received
0620 03:01:05.082148 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:05.082152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:05.082221 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:05.082299 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:05.083582 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.084315 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.085805 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.086212 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.086303 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:05.088522 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:05.088526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:05.088530 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:05.088557 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:05.091618 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:05.092224 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:05.092227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:05.092230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:05.092346 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:05.092664 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:05.092667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:05.092669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:05.092929 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:05.093096 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:05.093366 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 03:01:05.093526 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:01:05.098158 11702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:01:05.105072 11702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:01:05.130749 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:05.118194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.0.97.2:44099)
0620 03:01:05.118889 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:05.118899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:05.119876 (+ 977us) server_negotiation.cc:407] Connection header received
0620 03:01:05.120042 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:05.120053 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:05.120362 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:05.120576 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:01:05.121748 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.123863 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.127985 (+ 4122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.129554 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.129708 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:05.129911 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:05.130236 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 03:01:05.130400 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":407,"threads_started":1}
I20260620 03:01:05.130788 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:05.117982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44099 (local address 127.0.0.1:37814)
0620 03:01:05.119418 (+ 1436us) negotiation.cc:107] Waiting for socket to connect
0620 03:01:05.119467 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 03:01:05.119646 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:01:05.120915 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:01:05.120925 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:01:05.120980 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:01:05.121462 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:05.121484 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.124095 (+ 2611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.124102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:05.127774 (+ 3672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:05.127789 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.129766 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.129774 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:05.129978 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:05.130032 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 03:01:05.130164 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 03:01:05.130353 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"mutex_wait_us":1,"thread_start_us":1015,"threads_started":1}
I20260620 03:01:05.133407 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:05.118413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40683 (local address 127.0.0.1:47822)
0620 03:01:05.120882 (+ 2469us) negotiation.cc:107] Waiting for socket to connect
0620 03:01:05.120903 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:01:05.121031 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:01:05.125048 (+ 4017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:01:05.125057 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:01:05.125097 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:01:05.126293 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:05.126314 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.128715 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.128721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:05.130730 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:05.130750 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.132646 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.132653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:05.132838 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:05.132870 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 03:01:05.132935 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 03:01:05.133069 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2323,"thread_start_us":8208,"threads_started":1}
I20260620 03:01:05.134578 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:05.119411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47822 (local address 127.0.97.1:40683)
0620 03:01:05.121637 (+ 2226us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:05.121650 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:05.121684 (+ 34us) server_negotiation.cc:407] Connection header received
0620 03:01:05.121800 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:05.121811 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:05.122112 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:05.122330 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:01:05.127033 (+ 4703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.128562 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.130945 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.132388 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.133797 (+ 1409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:05.134024 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:05.134097 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 03:01:05.134258 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2004,"thread_start_us":1838,"threads_started":1}
I20260620 03:01:05.135063 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:05.119136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46049 (local address 127.0.0.1:52078)
0620 03:01:05.126930 (+ 7794us) negotiation.cc:107] Waiting for socket to connect
0620 03:01:05.126950 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:01:05.127054 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:01:05.128544 (+ 1490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:01:05.128550 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:01:05.128575 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:01:05.128997 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:05.129006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.131009 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.131017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:05.132765 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:05.132778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.134254 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.134260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:05.134425 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:05.134464 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 03:01:05.134578 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 03:01:05.134698 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7628,"mutex_wait_us":473,"thread_start_us":1001,"threads_started":1}
I20260620 03:01:05.135586 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:05.119418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52078 (local address 127.0.97.3:46049)
0620 03:01:05.127942 (+ 8524us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:05.127951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:05.127983 (+ 32us) server_negotiation.cc:407] Connection header received
0620 03:01:05.128078 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:05.128087 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:05.128331 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:05.129168 (+ 837us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:01:05.129382 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.130842 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.132925 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.134046 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.134201 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:05.134432 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:05.134615 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 03:01:05.134758 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8023,"thread_start_us":7870,"threads_started":1}
I20260620 03:01:05.135218 11994 tablet_service.cc:1467] Processing CreateTablet for tablet 793eee08bb354e0da1fb051e09e3a679 (DEFAULT_TABLE table=test-table [id=4d3bf75fcabc42d4a4df551e3845949a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:01:05.136794 11994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 793eee08bb354e0da1fb051e09e3a679. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:01:05.138305 11851 tablet_service.cc:1467] Processing CreateTablet for tablet 793eee08bb354e0da1fb051e09e3a679 (DEFAULT_TABLE table=test-table [id=4d3bf75fcabc42d4a4df551e3845949a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:01:05.139828 11851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 793eee08bb354e0da1fb051e09e3a679. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:01:05.145850 12128 tablet_service.cc:1467] Processing CreateTablet for tablet 793eee08bb354e0da1fb051e09e3a679 (DEFAULT_TABLE table=test-table [id=4d3bf75fcabc42d4a4df551e3845949a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:01:05.147392 12128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 793eee08bb354e0da1fb051e09e3a679. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:01:05.152225 12212 tablet_bootstrap.cc:492] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0: Bootstrap starting.
I20260620 03:01:05.154110 12213 tablet_bootstrap.cc:492] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33: Bootstrap starting.
I20260620 03:01:05.156041 12212 tablet_bootstrap.cc:654] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0: Neither blocks nor log segments found. Creating new log.
I20260620 03:01:05.157079 12212 log.cc:824] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0: Log is configured to *not* fsync() on all Append() calls
I20260620 03:01:05.158089 12213 tablet_bootstrap.cc:654] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33: Neither blocks nor log segments found. Creating new log.
I20260620 03:01:05.159281 12213 log.cc:824] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33: Log is configured to *not* fsync() on all Append() calls
I20260620 03:01:05.160285 12212 tablet_bootstrap.cc:492] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0: No bootstrap required, opened a new log
I20260620 03:01:05.160579 12212 ts_tablet_manager.cc:1397] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260620 03:01:05.162480 12213 tablet_bootstrap.cc:492] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33: No bootstrap required, opened a new log
I20260620 03:01:05.162751 12216 tablet_bootstrap.cc:492] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e: Bootstrap starting.
I20260620 03:01:05.163239 12213 ts_tablet_manager.cc:1397] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260620 03:01:05.165766 12216 tablet_bootstrap.cc:654] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e: Neither blocks nor log segments found. Creating new log.
I20260620 03:01:05.166706 12216 log.cc:824] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e: Log is configured to *not* fsync() on all Append() calls
I20260620 03:01:05.169191 12216 tablet_bootstrap.cc:492] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e: No bootstrap required, opened a new log
I20260620 03:01:05.169515 12216 ts_tablet_manager.cc:1397] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260620 03:01:05.171929 12212 raft_consensus.cc:348] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085958e9f7404fbe880f3c178afaddd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44099 } } peers { permanent_uuid: "4a3ddebe2b674820a3e77a86e4424b33" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "bec41d857fd44a48b3aeb9b2871d382e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46049 } }
I20260620 03:01:05.172379 12212 raft_consensus.cc:374] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:01:05.172480 12212 raft_consensus.cc:729] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085958e9f7404fbe880f3c178afaddd0, State: Initialized, Role: FOLLOWER
I20260620 03:01:05.173106 12212 consensus_queue.cc:260] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085958e9f7404fbe880f3c178afaddd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44099 } } peers { permanent_uuid: "4a3ddebe2b674820a3e77a86e4424b33" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "bec41d857fd44a48b3aeb9b2871d382e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46049 } }
I20260620 03:01:05.174268 12059 heartbeater.cc:503] Master 127.0.97.62:37509 was elected leader, sending a full tablet report...
I20260620 03:01:05.174059 12213 raft_consensus.cc:348] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085958e9f7404fbe880f3c178afaddd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44099 } } peers { permanent_uuid: "4a3ddebe2b674820a3e77a86e4424b33" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "bec41d857fd44a48b3aeb9b2871d382e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46049 } }
I20260620 03:01:05.174510 12213 raft_consensus.cc:374] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:01:05.174618 12213 raft_consensus.cc:729] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a3ddebe2b674820a3e77a86e4424b33, State: Initialized, Role: FOLLOWER
I20260620 03:01:05.174985 12212 ts_tablet_manager.cc:1428] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260620 03:01:05.175204 12213 consensus_queue.cc:260] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085958e9f7404fbe880f3c178afaddd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44099 } } peers { permanent_uuid: "4a3ddebe2b674820a3e77a86e4424b33" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "bec41d857fd44a48b3aeb9b2871d382e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46049 } }
I20260620 03:01:05.176807 12213 ts_tablet_manager.cc:1428] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 03:01:05.180776 12216 raft_consensus.cc:348] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085958e9f7404fbe880f3c178afaddd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44099 } } peers { permanent_uuid: "4a3ddebe2b674820a3e77a86e4424b33" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "bec41d857fd44a48b3aeb9b2871d382e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46049 } }
I20260620 03:01:05.181257 12216 raft_consensus.cc:374] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:01:05.181382 12216 raft_consensus.cc:729] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bec41d857fd44a48b3aeb9b2871d382e, State: Initialized, Role: FOLLOWER
I20260620 03:01:05.181980 12216 consensus_queue.cc:260] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085958e9f7404fbe880f3c178afaddd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44099 } } peers { permanent_uuid: "4a3ddebe2b674820a3e77a86e4424b33" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "bec41d857fd44a48b3aeb9b2871d382e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46049 } }
I20260620 03:01:05.182901 12193 heartbeater.cc:503] Master 127.0.97.62:37509 was elected leader, sending a full tablet report...
I20260620 03:01:05.183624 12216 ts_tablet_manager.cc:1428] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
W20260620 03:01:05.242404 12060 tablet.cc:2369] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:01:05.294584 12194 tablet.cc:2369] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:01:05.302127 11917 tablet.cc:2369] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:01:05.479167 12220 raft_consensus.cc:482] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:01:05.479552 12220 raft_consensus.cc:504] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085958e9f7404fbe880f3c178afaddd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44099 } } peers { permanent_uuid: "4a3ddebe2b674820a3e77a86e4424b33" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "bec41d857fd44a48b3aeb9b2871d382e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46049 } }
I20260620 03:01:05.481537 12220 leader_election.cc:288] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 085958e9f7404fbe880f3c178afaddd0 (127.0.97.2:44099), 4a3ddebe2b674820a3e77a86e4424b33 (127.0.97.1:40683)
I20260620 03:01:05.487411 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:05.481979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39983 (local address 127.0.97.2:44099)
0620 03:01:05.482155 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:05.482163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:05.482388 (+ 225us) server_negotiation.cc:407] Connection header received
0620 03:01:05.482488 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:05.482494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:05.482569 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:05.482675 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:01:05.483033 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.483796 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.485778 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.486539 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.486686 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:05.486781 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:05.487030 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 03:01:05.487154 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 03:01:05.487454 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:05.481849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44099 (local address 127.0.97.3:39983)
0620 03:01:05.482127 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 03:01:05.482147 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:01:05.482292 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:01:05.482693 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:01:05.482697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:01:05.482723 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:01:05.482890 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:05.482897 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.483911 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.483925 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:05.485643 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:05.485654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.486692 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.486698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:05.486875 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:05.486898 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 03:01:05.487005 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 03:01:05.487131 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260620 03:01:05.487995 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:05.482093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50373 (local address 127.0.97.1:40683)
0620 03:01:05.482321 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:05.482328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:05.482618 (+ 290us) server_negotiation.cc:407] Connection header received
0620 03:01:05.482770 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:05.482775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:05.482842 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:05.482937 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:01:05.483433 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.484220 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.486321 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.487093 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.487241 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:05.487365 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:05.487636 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 03:01:05.487755 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 03:01:05.488603 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:05.481974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40683 (local address 127.0.97.3:50373)
0620 03:01:05.482521 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 03:01:05.482534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 03:01:05.482661 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:01:05.482949 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:01:05.482953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:01:05.482974 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:01:05.483276 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:05.483283 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.484463 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.484467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:05.486174 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:05.486185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.487287 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.487293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:05.487461 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:05.487490 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 03:01:05.488156 (+ 666us) client_negotiation.cc:240] Negotiation successful
0620 03:01:05.488273 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":276,"threads_started":1}
I20260620 03:01:05.489219 12014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793eee08bb354e0da1fb051e09e3a679" candidate_uuid: "bec41d857fd44a48b3aeb9b2871d382e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "085958e9f7404fbe880f3c178afaddd0" is_pre_election: true
I20260620 03:01:05.489604 11871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793eee08bb354e0da1fb051e09e3a679" candidate_uuid: "bec41d857fd44a48b3aeb9b2871d382e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a3ddebe2b674820a3e77a86e4424b33" is_pre_election: true
I20260620 03:01:05.489895 12014 raft_consensus.cc:2454] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bec41d857fd44a48b3aeb9b2871d382e in term 0.
I20260620 03:01:05.490129 11871 raft_consensus.cc:2454] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bec41d857fd44a48b3aeb9b2871d382e in term 0.
I20260620 03:01:05.490610 12083 leader_election.cc:302] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085958e9f7404fbe880f3c178afaddd0, bec41d857fd44a48b3aeb9b2871d382e; no voters:
I20260620 03:01:05.491007 12220 raft_consensus.cc:2781] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:01:05.491153 12220 raft_consensus.cc:482] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:01:05.491250 12220 raft_consensus.cc:3037] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:01:05.493346 12220 raft_consensus.cc:504] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085958e9f7404fbe880f3c178afaddd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44099 } } peers { permanent_uuid: "4a3ddebe2b674820a3e77a86e4424b33" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "bec41d857fd44a48b3aeb9b2871d382e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46049 } }
I20260620 03:01:05.493976 12220 leader_election.cc:288] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [CANDIDATE]: Term 1 election: Requested vote from peers 085958e9f7404fbe880f3c178afaddd0 (127.0.97.2:44099), 4a3ddebe2b674820a3e77a86e4424b33 (127.0.97.1:40683)
I20260620 03:01:05.494387 12014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793eee08bb354e0da1fb051e09e3a679" candidate_uuid: "bec41d857fd44a48b3aeb9b2871d382e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "085958e9f7404fbe880f3c178afaddd0"
I20260620 03:01:05.494431 11871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793eee08bb354e0da1fb051e09e3a679" candidate_uuid: "bec41d857fd44a48b3aeb9b2871d382e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a3ddebe2b674820a3e77a86e4424b33"
I20260620 03:01:05.494592 12014 raft_consensus.cc:3037] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:01:05.494619 11871 raft_consensus.cc:3037] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:01:05.496764 12014 raft_consensus.cc:2454] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bec41d857fd44a48b3aeb9b2871d382e in term 1.
I20260620 03:01:05.497105 12083 leader_election.cc:302] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085958e9f7404fbe880f3c178afaddd0, bec41d857fd44a48b3aeb9b2871d382e; no voters:
I20260620 03:01:05.497385 12220 raft_consensus.cc:2781] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:01:05.497718 11871 raft_consensus.cc:2454] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bec41d857fd44a48b3aeb9b2871d382e in term 1.
I20260620 03:01:05.498345 12220 raft_consensus.cc:686] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [term 1 LEADER]: Becoming Leader. State: Replica: bec41d857fd44a48b3aeb9b2871d382e, State: Running, Role: LEADER
I20260620 03:01:05.498801 12220 consensus_queue.cc:237] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085958e9f7404fbe880f3c178afaddd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44099 } } peers { permanent_uuid: "4a3ddebe2b674820a3e77a86e4424b33" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "bec41d857fd44a48b3aeb9b2871d382e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46049 } }
I20260620 03:01:05.503535 11703 catalog_manager.cc:5495] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e reported cstate change: term changed from 0 to 1, leader changed from <none> to bec41d857fd44a48b3aeb9b2871d382e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "bec41d857fd44a48b3aeb9b2871d382e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085958e9f7404fbe880f3c178afaddd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a3ddebe2b674820a3e77a86e4424b33" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bec41d857fd44a48b3aeb9b2871d382e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46049 } health_report { overall_health: HEALTHY } } }
I20260620 03:01:05.585691 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:05.580372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52080 (local address 127.0.97.3:46049)
0620 03:01:05.580591 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:05.580598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:05.580642 (+ 44us) server_negotiation.cc:407] Connection header received
0620 03:01:05.580757 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:05.580761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:05.580842 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:05.580984 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:01:05.581365 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.582148 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.583553 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.583939 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.584085 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:05.584585 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:01:05.585082 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:01:05.585238 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:05.585295 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 03:01:05.585369 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 03:01:05.593762 12014 raft_consensus.cc:1261] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0 [term 1 FOLLOWER]: Refusing update from remote peer bec41d857fd44a48b3aeb9b2871d382e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:01:05.593779 11871 raft_consensus.cc:1261] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33 [term 1 FOLLOWER]: Refusing update from remote peer bec41d857fd44a48b3aeb9b2871d382e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:01:05.594749 12222 consensus_queue.cc:1035] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [LEADER]: Connected to new peer: Peer: permanent_uuid: "085958e9f7404fbe880f3c178afaddd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:01:05.595084 12220 consensus_queue.cc:1035] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3ddebe2b674820a3e77a86e4424b33" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:01:05.607151 12227 mvcc.cc:205] Tried to move back new op lower bound from 7298762611062091776 to 7298762610687033344. Current Snapshot: MvccSnapshot[applied={T|T < 7298762611062091776}]
I20260620 03:01:05.622059 11702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:01:05.622370 11702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:01:05.626024 11702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7377f6626244abba729f49807f4711e: Sending DeleteTablet for 3 replicas of tablet 793eee08bb354e0da1fb051e09e3a679
I20260620 03:01:05.627220 11994 tablet_service.cc:1514] Processing DeleteTablet for tablet 793eee08bb354e0da1fb051e09e3a679 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:01:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37814
I20260620 03:01:05.627228 11851 tablet_service.cc:1514] Processing DeleteTablet for tablet 793eee08bb354e0da1fb051e09e3a679 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:01:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47822
I20260620 03:01:05.627367 12128 tablet_service.cc:1514] Processing DeleteTablet for tablet 793eee08bb354e0da1fb051e09e3a679 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:01:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52078
I20260620 03:01:05.629949 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 11783
I20260620 03:01:05.640481 12235 tablet_replica.cc:330] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e: stopping tablet replica
I20260620 03:01:05.641649 12235 raft_consensus.cc:2229] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:01:05.643131 12235 raft_consensus.cc:2258] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:01:05.647132 12235 ts_tablet_manager.cc:1905] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:01:05.648221 12234 tablet_replica.cc:330] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33: stopping tablet replica
I20260620 03:01:05.648833 12236 tablet_replica.cc:330] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0: stopping tablet replica
I20260620 03:01:05.649437 12234 raft_consensus.cc:2229] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:01:05.649967 12236 raft_consensus.cc:2229] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:01:05.650559 12236 raft_consensus.cc:2258] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:01:05.651072 12234 raft_consensus.cc:2258] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:01:05.653785 12236 ts_tablet_manager.cc:1905] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:01:05.655790 12235 ts_tablet_manager.cc:1918] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:01:05.656005 12235 log.cc:1196] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/793eee08bb354e0da1fb051e09e3a679
Jun 20 03:01:05 dist-test-slave-f1j0 krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:01:05.656533 12235 ts_tablet_manager.cc:1939] T 793eee08bb354e0da1fb051e09e3a679 P bec41d857fd44a48b3aeb9b2871d382e: Deleting consensus metadata
I20260620 03:01:05.656708 12234 ts_tablet_manager.cc:1905] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:01:05.658877 11690 catalog_manager.cc:4841] TS bec41d857fd44a48b3aeb9b2871d382e (127.0.97.3:46049): tablet 793eee08bb354e0da1fb051e09e3a679 (table test-table [id=4d3bf75fcabc42d4a4df551e3845949a]) successfully deleted
I20260620 03:01:05.660885 12236 ts_tablet_manager.cc:1918] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:01:05.661091 12236 log.cc:1196] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/793eee08bb354e0da1fb051e09e3a679
I20260620 03:01:05.661679 12236 ts_tablet_manager.cc:1939] T 793eee08bb354e0da1fb051e09e3a679 P 085958e9f7404fbe880f3c178afaddd0: Deleting consensus metadata
I20260620 03:01:05.663452 11689 catalog_manager.cc:4841] TS 085958e9f7404fbe880f3c178afaddd0 (127.0.97.2:44099): tablet 793eee08bb354e0da1fb051e09e3a679 (table test-table [id=4d3bf75fcabc42d4a4df551e3845949a]) successfully deleted
I20260620 03:01:05.664515 12234 ts_tablet_manager.cc:1918] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:01:05.664669 12234 log.cc:1196] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/793eee08bb354e0da1fb051e09e3a679
I20260620 03:01:05.665169 12234 ts_tablet_manager.cc:1939] T 793eee08bb354e0da1fb051e09e3a679 P 4a3ddebe2b674820a3e77a86e4424b33: Deleting consensus metadata
I20260620 03:01:05.666913 11687 catalog_manager.cc:4841] TS 4a3ddebe2b674820a3e77a86e4424b33 (127.0.97.1:40683): tablet 793eee08bb354e0da1fb051e09e3a679 (table test-table [id=4d3bf75fcabc42d4a4df551e3845949a]) successfully deleted
I20260620 03:01:05.667616 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:05.631710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47826 (local address 127.0.97.1:40683)
0620 03:01:05.631933 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:05.631942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:05.639514 (+ 7572us) server_negotiation.cc:407] Connection header received
0620 03:01:05.639776 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:05.639785 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:05.639930 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:05.640060 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:05.642327 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.643842 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.646602 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.647535 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.647659 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:05.660294 (+ 12635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:05.660309 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:05.660326 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:05.660370 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:05.664412 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:05.665297 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:05.665306 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:05.665309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:05.665478 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:05.665968 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:05.665973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:05.665976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:05.666427 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:05.666720 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:05.667048 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 03:01:05.667281 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 03:01:05.668277 11911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:01:05.772243 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 11783
I20260620 03:01:05.790374 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 11922
Jun 20 03:01:05 dist-test-slave-f1j0 krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:01:05.807679 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:05.791696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37826 (local address 127.0.97.2:44099)
0620 03:01:05.791909 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:05.791916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:05.791937 (+ 21us) server_negotiation.cc:407] Connection header received
0620 03:01:05.792002 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:05.792008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:05.792088 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:05.792201 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:05.793464 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.794701 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.796023 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.796671 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.796818 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:05.801080 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:05.801101 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:05.801120 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:05.801168 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:05.804427 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:05.805251 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:05.805258 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:05.805260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:05.805468 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:05.805948 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:05.805955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:05.805960 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:05.806592 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:05.806801 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:05.807150 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 03:01:05.807340 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 03:01:05.808364 12054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:01:05.898121 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 11922
I20260620 03:01:05.915640 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 12065
Jun 20 03:01:05 dist-test-slave-f1j0 krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:01:05.933617 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:05.916547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52088 (local address 127.0.97.3:46049)
0620 03:01:05.916741 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:05.916748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:05.916778 (+ 30us) server_negotiation.cc:407] Connection header received
0620 03:01:05.916913 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:05.916919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:05.917000 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:05.917137 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:05.918487 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.919751 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.921059 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:05.921782 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:05.921935 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:05.926767 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:05.926785 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:05.926805 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:05.926868 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:05.930842 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:05.931491 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:05.931502 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:05.931506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:05.931683 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:05.932022 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:05.932027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:05.932031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:05.932494 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:05.932835 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:05.932988 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 03:01:05.933212 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 03:01:05.934361 12188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:01:06.057907 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 12065
I20260620 03:01:06.077404 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 11671
I20260620 03:01:06.078645 11733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:01:06.193989 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 11671
I20260620 03:01:06.214807 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 03:01:06.550634 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 03:01:06.550873 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 03:01:06.896658 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 03:01:06.896 UTC [11257] LOG: received smart shutdown request
2026-06-20 03:01:06.900 UTC [11257] LOG: background worker "logical replication launcher" (PID 11266) exited with exit code 1
2026-06-20 03:01:06.900 UTC [11261] LOG: shutting down
2026-06-20 03:01:07.051 UTC [11257] LOG: database system is shut down
2026-06-20T03:01:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49352 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:01:07 dist-test-slave-f1j0 krb5kdc[12250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:01:07 dist-test-slave-f1j0 krb5kdc[12250](info): set up 2 sockets
Jun 20 03:01:07 dist-test-slave-f1j0 krb5kdc[12250](info): commencing operation
krb5kdc: starting...
W20260620 03:01:08.278333 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:01:08 dist-test-slave-f1j0 krb5kdc[12250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:01:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:01:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 03:01:08.600329 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpRaC98/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/postgres -l logfile start
2026-06-20 03:01:11.004 UTC [12280] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 03:01:11.004 UTC [12280] LOG: listening on IPv4 address "127.0.97.20", port 43233
2026-06-20 03:01:11.007 UTC [12280] LOG: listening on Unix socket "/tmp/.s.PGSQL.43233"
2026-06-20 03:01:11.031 UTC [12283] LOG: database system was shut down at 2026-06-20 03:01:09 UTC
2026-06-20 03:01:11.035 UTC [12280] LOG: database system is ready to accept connections
I20260620 03:01:12.013499 388 mini_postgres.cc:96] Postgres bound to 43233
2026-06-20 03:01:12.030 UTC [12291] FATAL: database "slave" does not exist
127.0.97.20:43233 - accepting connections
I20260620 03:01:12.031394 388 mini_ranger.cc:161] Starting Ranger...
I20260620 03:01:12.062834 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 03:01:12.534997 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 03:01:12.535128 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ranger-admin
2026-06-20 03:01:12,824 [I] DB FLAVOR :POSTGRES
2026-06-20 03:01:12,824 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 03:01:12,825 [I] Checking connection..
2026-06-20 03:01:12,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 03:01:13,086 [I] Checking connection passed.
2026-06-20 03:01:13,086 [I] --------- Verifying version history table ---------
2026-06-20 03:01:13,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 03:01:13,327 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 03:01:13,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 03:01:13,593 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 03:01:13,593 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 03:01:13,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 03:01:13.838 UTC [12355] WARNING: there is no transaction in progress
2026-06-20 03:01:13,844 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 03:01:13,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 03:01:14,075 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 03:01:14,076 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 03:01:14,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 03:01:14,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 03:01:14,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f1j0.c.gcp-upstream.internal','N') ;"
2026-06-20 03:01:14,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 03:01:15,060 [I] Table x_portal_user does not exist in database ranger
2026-06-20 03:01:15,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 03:01:15,303 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 03:01:15,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 03:01:15,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 03:01:15,793 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 03:01:15,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 03:01:16.757 UTC [12463] WARNING: there is no transaction in progress
2026-06-20 03:01:16.774 UTC [12463] WARNING: there is no transaction in progress
2026-06-20 03:01:16.791 UTC [12463] WARNING: there is no transaction in progress
2026-06-20 03:01:16.806 UTC [12463] WARNING: there is no transaction in progress
2026-06-20 03:01:16.822 UTC [12463] WARNING: there is no transaction in progress
2026-06-20 03:01:16.843 UTC [12463] WARNING: there is no transaction in progress
2026-06-20 03:01:16.907 UTC [12463] WARNING: there is no transaction in progress
2026-06-20 03:01:16.914 UTC [12463] WARNING: there is no transaction in progress
2026-06-20 03:01:16.929 UTC [12463] WARNING: there is no transaction in progress
2026-06-20 03:01:16.939 UTC [12463] WARNING: there is no transaction in progress
2026-06-20 03:01:16.949 UTC [12463] WARNING: there is no transaction in progress
2026-06-20 03:01:16.962 UTC [12463] WARNING: there is no transaction in progress
2026-06-20 03:01:16.972 UTC [12463] WARNING: there is no transaction in progress
2026-06-20 03:01:16.979 UTC [12463] WARNING: there is no transaction in progress
2026-06-20 03:01:16.986 UTC [12463] WARNING: there is no transaction in progress
2026-06-20 03:01:16.993 UTC [12463] WARNING: there is no transaction in progress
2026-06-20 03:01:17,604 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 03:01:17,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 03:01:17,846 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 03:01:17,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 03:01:18,104 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 03:01:18,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 03:01:18,352 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 03:01:18,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 03:01:18,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 03:01:18,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f1j0.c.gcp-upstream.internal';"
2026-06-20 03:01:19,087 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 03:01:19,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 03:01:19,335 [I] DB_PATCHES have already been applied
I20260620 03:01:19.341542 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpRaC98/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 03:01:19.341718 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 03:01:19.356288 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:47777
Jun 20, 2026 3:01:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:01:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:01:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 3:01:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 3:01:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 3:01:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 3:01:20 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 03:01:20.543462 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.512s user 0.002s sys 0.041s
Jun 20, 2026 3:01:41 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 3:01:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 3:01:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 3:01:41 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 3:01:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 3:01:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 3:01:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 3:01:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 3:01:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 3:01:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 3:01:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 3:01:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 3:01:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 3:01:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 3:01:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 3:01:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 3:01:41 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 3:01:41 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 03:01:42.542057 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 03:01:42 dist-test-slave-f1j0 krb5kdc[12250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924502, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 03:01:42.702706 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 03:01:42.722757 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 03:01:43.553663 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 03:01:43.553781 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ranger-kms
I20260620 03:01:43.553855 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:43233
2026-06-20 03:01:43,918 [I] DB FLAVOR :POSTGRES
2026-06-20 03:01:43,919 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 03:01:43,919 [I] Checking connection
2026-06-20 03:01:43,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 03:01:44,155 [I] connection success
2026-06-20 03:01:44,155 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 03:01:44,155 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 03:01:44,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 03:01:44,419 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 03:01:44,420 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 03:01:44,420 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 03:01:44,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43233/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 03:01:44,693 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 03:01:45.180533 388 mini_ranger_kms.cc:310] Created kms service
I20260620 03:01:45.358876 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 03:01:45.402863 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 03:01:45 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924502, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:01:45.819990 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 03:01:45.820325 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpRaC98/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpRaC98/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 03:01:45.820380 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 03:01:45.837433 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12643
I20260620 03:01:45.837584 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:41007
Jun 20, 2026 3:01:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:01:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:01:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 3:01:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 3:01:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 3:01:47 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 03:01:47 dist-test-slave-f1j0 krb5kdc[12250](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781924507, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:01:48.122644 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.420s user 0.071s sys 0.309s
I20260620 03:01:48.122905 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 03:01:48.123088 388 mini_ranger_kms.cc:401] 127.0.97.20:41007/kms/v1/keys
Jun 20 03:01:48 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924507, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 03:01:48 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924507, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 03:01:50 dist-test-slave-f1j0 krb5kdc[12250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:01:51.029480 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39679
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39679
--ranger_config_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41007/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:01:51.383934 12694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:01:51.384373 12694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:01:51.384531 12694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:01:51.395967 12694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:01:51.396126 12694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:01:51.396194 12694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:01:51.396263 12694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:01:51.396333 12694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:01:51.412415 12694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:41007/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39679
--ranger_config_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:01:51.414175 12694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:01:51.415961 12694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:01:51.430259 12700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:51.430289 12699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:51.430784 12702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:01:51.431581 12694 server_base.cc:1034] running on GCE node
I20260620 03:01:51.432446 12694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:01:51.434063 12694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:01:51.435307 12694 hybrid_clock.cc:648] HybridClock initialized: now 1781924511435268 us; error 38 us; skew 500 ppm
I20260620 03:01:51.435662 12694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:01:51 dist-test-slave-f1j0 krb5kdc[12250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:01:51.438871 12694 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:01:51.440430 12694 webserver.cc:460] Webserver started at http://127.0.97.62:38273/ using document root <none> and password file <none>
I20260620 03:01:51.441098 12694 fs_manager.cc:359] Metadata directory not provided
I20260620 03:01:51.441191 12694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:01:51.441532 12694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 03:01:51 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:01:51.505204 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "2f464c6ca77e4a298dd81f9a78794dff"
format_stamp: "Formatted at 2026-06-20 03:01:51 on dist-test-slave-f1j0"
server_key: "242a53f2419a934d925e769c1524cbf8"
server_key_iv: "19714181e648f046f367cb874705ae53"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:51.506134 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "2f464c6ca77e4a298dd81f9a78794dff"
format_stamp: "Formatted at 2026-06-20 03:01:51 on dist-test-slave-f1j0"
server_key: "242a53f2419a934d925e769c1524cbf8"
server_key_iv: "19714181e648f046f367cb874705ae53"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:51.511723 12694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:01:51.532537 12709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:51.534261 12694 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 03:01:51.534479 12694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "2f464c6ca77e4a298dd81f9a78794dff"
format_stamp: "Formatted at 2026-06-20 03:01:51 on dist-test-slave-f1j0"
server_key: "242a53f2419a934d925e769c1524cbf8"
server_key_iv: "19714181e648f046f367cb874705ae53"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:51.534729 12694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:01:51.552095 12694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:01:51.556445 12694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:01:51.556762 12694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:01:51.576735 12694 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39679
I20260620 03:01:51.576757 12760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39679 every 8 connection(s)
I20260620 03:01:51.578280 12694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:01:51.582237 12761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:01:51.586879 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 12694
I20260620 03:01:51.587126 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:01:51.588025 388 external_mini_cluster.cc:1428] Setting key 0e0079d86bb0b967b8745cb63f0ee1d2
I20260620 03:01:51.596415 12761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff: Bootstrap starting.
I20260620 03:01:51.602038 12761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff: Neither blocks nor log segments found. Creating new log.
I20260620 03:01:51.603641 12761 log.cc:824] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff: Log is configured to *not* fsync() on all Append() calls
I20260620 03:01:51.607307 12761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff: No bootstrap required, opened a new log
I20260620 03:01:51.616410 12761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f464c6ca77e4a298dd81f9a78794dff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39679 } }
Jun 20 03:01:51 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:01:51.616912 12761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:01:51.617049 12761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f464c6ca77e4a298dd81f9a78794dff, State: Initialized, Role: FOLLOWER
I20260620 03:01:51.617830 12761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f464c6ca77e4a298dd81f9a78794dff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39679 } }
I20260620 03:01:51.618148 12761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:01:51.618311 12761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:01:51.618532 12761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:01:51.621255 12761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f464c6ca77e4a298dd81f9a78794dff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39679 } }
I20260620 03:01:51.621899 12761 leader_election.cc:302] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f464c6ca77e4a298dd81f9a78794dff; no voters:
I20260620 03:01:51.623839 12761 leader_election.cc:288] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:01:51.623951 12766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:01:51.629345 12764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:51.590334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45664 (local address 127.0.97.62:39679)
0620 03:01:51.602201 (+ 11867us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:51.602222 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:51.602659 (+ 437us) server_negotiation.cc:407] Connection header received
0620 03:01:51.605228 (+ 2569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:51.605292 (+ 64us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:51.605840 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:51.606310 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:51.607832 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:51.609909 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:51.611775 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:51.612931 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:51.613135 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:51.618142 (+ 5007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:51.618152 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:51.618174 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:51.618223 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:51.622221 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:51.624296 (+ 2075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:51.624305 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:51.624314 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:51.624501 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:51.624992 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:51.624997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:51.625001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:51.625781 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:51.627155 (+ 1374us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:51.627450 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 03:01:51.627902 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11588,"thread_start_us":11330,"threads_started":1}
I20260620 03:01:51.633185 12761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:01:51.635195 12766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [term 1 LEADER]: Becoming Leader. State: Replica: 2f464c6ca77e4a298dd81f9a78794dff, State: Running, Role: LEADER
I20260620 03:01:51.635908 12766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f464c6ca77e4a298dd81f9a78794dff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39679 } }
I20260620 03:01:51.639475 12767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f464c6ca77e4a298dd81f9a78794dff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f464c6ca77e4a298dd81f9a78794dff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39679 } } }
I20260620 03:01:51.639461 12768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f464c6ca77e4a298dd81f9a78794dff. Latest consensus state: current_term: 1 leader_uuid: "2f464c6ca77e4a298dd81f9a78794dff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f464c6ca77e4a298dd81f9a78794dff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39679 } } }
I20260620 03:01:51.640105 12767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [sys.catalog]: This master's current role is: LEADER
I20260620 03:01:51.642058 12768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [sys.catalog]: This master's current role is: LEADER
I20260620 03:01:51.649215 12775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:01:51.655679 12775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:01:51.668475 12775 catalog_manager.cc:1347] Generated new cluster ID: 372755a267174c9da5a3e14590734e99
I20260620 03:01:51.668601 12775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:01:51.678754 12761 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 03:01:51.713529 12775 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:01:51.715123 12775 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:01:51.726739 12775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff: Generated new TSK 0
I20260620 03:01:51.727993 12775 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:01:52.585990 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39679
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:01:52 dist-test-slave-f1j0 krb5kdc[12250](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781924512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 03:01:53.053802 12806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:01:53.054188 12806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:01:53.054387 12806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:01:53.065964 12806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:01:53.066149 12806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:01:53.066361 12806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:01:53.082051 12806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:41007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:01:53.083958 12806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:01:53.085712 12806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:01:53.102869 12819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:53.102989 12816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:53.102720 12817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:01:53.104067 12806 server_base.cc:1034] running on GCE node
I20260620 03:01:53.104751 12806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:01:53.105859 12806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:01:53.107072 12806 hybrid_clock.cc:648] HybridClock initialized: now 1781924513107013 us; error 57 us; skew 500 ppm
I20260620 03:01:53.107406 12806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:01:53 dist-test-slave-f1j0 krb5kdc[12250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:01:53.110836 12806 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:01:53.112494 12806 webserver.cc:460] Webserver started at http://127.0.97.1:43849/ using document root <none> and password file <none>
I20260620 03:01:53.113201 12806 fs_manager.cc:359] Metadata directory not provided
I20260620 03:01:53.113358 12806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:01:53.113734 12806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 03:01:53 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:01:53.145617 12806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "6ee2b5240a8b4ec0a65bfc4bf25d17ef"
format_stamp: "Formatted at 2026-06-20 03:01:53 on dist-test-slave-f1j0"
server_key: "8cf6b4af28de1256a56db94910bd1e92"
server_key_iv: "94048e52a802f2e6c3f84d07d6aaa097"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:53.146800 12806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6ee2b5240a8b4ec0a65bfc4bf25d17ef"
format_stamp: "Formatted at 2026-06-20 03:01:53 on dist-test-slave-f1j0"
server_key: "8cf6b4af28de1256a56db94910bd1e92"
server_key_iv: "94048e52a802f2e6c3f84d07d6aaa097"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:53.155069 12806 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260620 03:01:53.190384 12826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:53.192641 12806 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.006s sys 0.009s
I20260620 03:01:53.192977 12806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "6ee2b5240a8b4ec0a65bfc4bf25d17ef"
format_stamp: "Formatted at 2026-06-20 03:01:53 on dist-test-slave-f1j0"
server_key: "8cf6b4af28de1256a56db94910bd1e92"
server_key_iv: "94048e52a802f2e6c3f84d07d6aaa097"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:53.193338 12806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:01:53.259446 12806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:01:53.264997 12806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:01:53.265388 12806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:01:53.272771 12806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:01:53.280834 12806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:01:53.281009 12806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:53.281198 12806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:01:53.281299 12806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 03:01:53 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:01:53.365602 12806 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39043
I20260620 03:01:53.365804 12938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39043 every 8 connection(s)
I20260620 03:01:53.367705 12806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:01:53.377820 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 12806
I20260620 03:01:53.378041 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:01:53.378890 388 external_mini_cluster.cc:1428] Setting key a6dc9e8502f4387c8f4793633a9734b8
Jun 20 03:01:53 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 03:01:53.420524 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:53.376454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32955 (local address 127.0.97.62:39679)
0620 03:01:53.396294 (+ 19840us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:53.396306 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:53.397738 (+ 1432us) server_negotiation.cc:407] Connection header received
0620 03:01:53.400805 (+ 3067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:53.400812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:53.400934 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:53.401069 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:53.403360 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:53.404565 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:53.406970 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:53.407636 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:53.407738 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:53.412588 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:53.412598 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:53.412605 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:53.412649 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:53.416364 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:53.417470 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:53.417476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:53.417481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:53.417716 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:53.418324 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:53.418331 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:53.418334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:53.418704 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:53.418860 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:53.419879 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0620 03:01:53.420109 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19593,"thread_start_us":19844,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 03:01:53.423105 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:53.377275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39679 (local address 127.0.97.1:32955)
0620 03:01:53.397070 (+ 19795us) negotiation.cc:107] Waiting for socket to connect
0620 03:01:53.397131 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 03:01:53.400428 (+ 3297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:01:53.401554 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:01:53.401598 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:01:53.402266 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:01:53.403120 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:53.403147 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:53.404762 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:53.404768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:53.406797 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:53.406811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:53.408119 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:53.408127 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:53.408444 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:53.409338 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:01:53.409369 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:01:53.412283 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:01:53.416640 (+ 4357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:01:53.416650 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:01:53.416671 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:01:53.417327 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:01:53.417884 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:01:53.417889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:01:53.417893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:01:53.418174 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:01:53.418958 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:01:53.418967 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:01:53.419425 (+ 458us) client_negotiation.cc:769] Sending connection context
0620 03:01:53.420764 (+ 1339us) client_negotiation.cc:240] Negotiation successful
0620 03:01:53.421505 (+ 741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19465,"thread_start_us":19196,"threads_started":1}
I20260620 03:01:53.429081 12939 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39679
I20260620 03:01:53.429452 12939 heartbeater.cc:464] Registering TS with master...
I20260620 03:01:53.430419 12939 heartbeater.cc:511] Master 127.0.97.62:39679 requested a full tablet report, sending...
I20260620 03:01:53.432943 12726 ts_manager.cc:194] Registered new tserver with Master: 6ee2b5240a8b4ec0a65bfc4bf25d17ef (127.0.97.1:39043)
I20260620 03:01:53.434993 12726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:32955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:01:53.464866 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39679
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:01:53 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 03:01:53.954604 12945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:01:53.954969 12945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:01:53.955090 12945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:01:53.965525 12945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:01:53.965749 12945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:01:53.965942 12945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:01:53.980222 12945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:41007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:01:53.981978 12945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:01:53.983703 12945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:01:53.998901 12959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:53.999310 12962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:53.999605 12960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:01:53.999785 12945 server_base.cc:1034] running on GCE node
I20260620 03:01:54.000501 12945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:01:54.001498 12945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:01:54.002734 12945 hybrid_clock.cc:648] HybridClock initialized: now 1781924514002673 us; error 54 us; skew 500 ppm
I20260620 03:01:54.003088 12945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:01:54 dist-test-slave-f1j0 krb5kdc[12250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:01:54.006538 12945 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:01:54.008062 12945 webserver.cc:460] Webserver started at http://127.0.97.2:36415/ using document root <none> and password file <none>
I20260620 03:01:54.008739 12945 fs_manager.cc:359] Metadata directory not provided
I20260620 03:01:54.008881 12945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:01:54.009222 12945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 03:01:54 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:01:54.024166 12945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "28d3927fd978440dbea6a4d20ef45851"
format_stamp: "Formatted at 2026-06-20 03:01:54 on dist-test-slave-f1j0"
server_key: "a92f6ad6f64672fdc13712c76e65f242"
server_key_iv: "0f7d354bb90df99968e277c5632c7b41"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:54.024991 12945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "28d3927fd978440dbea6a4d20ef45851"
format_stamp: "Formatted at 2026-06-20 03:01:54 on dist-test-slave-f1j0"
server_key: "a92f6ad6f64672fdc13712c76e65f242"
server_key_iv: "0f7d354bb90df99968e277c5632c7b41"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:54.030542 12945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 03:01:54.054270 12969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:54.056355 12945 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.009s sys 0.000s
I20260620 03:01:54.056555 12945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "28d3927fd978440dbea6a4d20ef45851"
format_stamp: "Formatted at 2026-06-20 03:01:54 on dist-test-slave-f1j0"
server_key: "a92f6ad6f64672fdc13712c76e65f242"
server_key_iv: "0f7d354bb90df99968e277c5632c7b41"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:54.056826 12945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:01:54.084364 12945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:01:54.089717 12945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:01:54.090057 12945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:01:54.091270 12945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:01:54.093071 12945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:01:54.093170 12945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:54.093348 12945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:01:54.093441 12945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:54.151643 12945 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37789
I20260620 03:01:54.151686 13081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37789 every 8 connection(s)
I20260620 03:01:54.153348 12945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:01:54.157431 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 12945
I20260620 03:01:54.157754 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:01:54.158397 388 external_mini_cluster.cc:1428] Setting key 830540fcdc6c58d7eb1d38ed444fd868
Jun 20 03:01:54 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:01:54.204094 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:54.164987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48397 (local address 127.0.97.62:39679)
0620 03:01:54.182165 (+ 17178us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:54.182176 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:54.183165 (+ 989us) server_negotiation.cc:407] Connection header received
0620 03:01:54.185405 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:54.185410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:54.185521 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:54.185669 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:54.187888 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:54.189050 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:54.191459 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:54.192113 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:54.192209 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:54.196684 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:54.196691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:54.196696 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:54.196744 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:54.200423 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:54.201387 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:54.201393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:54.201396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:54.201536 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:54.202079 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:54.202084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:54.202087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:54.202429 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:54.202550 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:54.203497 (+ 947us) server_negotiation.cc:299] Negotiation successful
0620 03:01:54.203720 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16950,"thread_start_us":17187,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 03:01:54.206499 13084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:54.162540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39679 (local address 127.0.97.2:48397)
0620 03:01:54.182919 (+ 20379us) negotiation.cc:107] Waiting for socket to connect
0620 03:01:54.182989 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 03:01:54.185062 (+ 2073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:01:54.186145 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:01:54.186206 (+ 61us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:01:54.186838 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:01:54.187671 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:54.187697 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:54.189271 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:54.189277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:54.191286 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:54.191300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:54.192586 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:54.192592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:54.192888 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:54.193706 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:01:54.193736 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:01:54.196420 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:01:54.200625 (+ 4205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:01:54.200635 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:01:54.200655 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:01:54.201236 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:01:54.201692 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:01:54.201697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:01:54.201701 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:01:54.201957 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:01:54.202654 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:01:54.202663 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:01:54.203076 (+ 413us) client_negotiation.cc:769] Sending connection context
0620 03:01:54.204585 (+ 1509us) client_negotiation.cc:240] Negotiation successful
0620 03:01:54.205117 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20032,"thread_start_us":19793,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 03:01:54.208417 13082 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39679
I20260620 03:01:54.208711 13082 heartbeater.cc:464] Registering TS with master...
I20260620 03:01:54.209489 13082 heartbeater.cc:511] Master 127.0.97.62:39679 requested a full tablet report, sending...
I20260620 03:01:54.211076 12726 ts_manager.cc:194] Registered new tserver with Master: 28d3927fd978440dbea6a4d20ef45851 (127.0.97.2:37789)
I20260620 03:01:54.212073 12726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:01:54.240993 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39679
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:01:54.439162 12939 heartbeater.cc:503] Master 127.0.97.62:39679 was elected leader, sending a full tablet report...
W20260620 03:01:54.646450 13088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:01:54.646716 13088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:01:54.646914 13088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:01:54.655347 13088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:01:54.655453 13088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:01:54.655649 13088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:01:54.665692 13088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:41007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:01:54.666970 13088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:01:54.668238 13088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:01:54.680902 13093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:54.680969 13096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:54.680997 13094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:01:54.682018 13088 server_base.cc:1034] running on GCE node
I20260620 03:01:54.682638 13088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:01:54.683656 13088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:01:54.684900 13088 hybrid_clock.cc:648] HybridClock initialized: now 1781924514684861 us; error 64 us; skew 500 ppm
I20260620 03:01:54.685286 13088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:01:54 dist-test-slave-f1j0 krb5kdc[12250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:01:54.688661 13088 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:01:54.690158 13088 webserver.cc:460] Webserver started at http://127.0.97.3:43213/ using document root <none> and password file <none>
I20260620 03:01:54.690838 13088 fs_manager.cc:359] Metadata directory not provided
I20260620 03:01:54.690979 13088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:01:54.691287 13088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 03:01:54 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:01:54.708545 13088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "2baa988c223b4c6aad408da9fd0c92b0"
format_stamp: "Formatted at 2026-06-20 03:01:54 on dist-test-slave-f1j0"
server_key: "7471facee7558a8bdf33060c2ffb4f55"
server_key_iv: "efae62415533def68454351cf6d437da"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:54.709652 13088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2baa988c223b4c6aad408da9fd0c92b0"
format_stamp: "Formatted at 2026-06-20 03:01:54 on dist-test-slave-f1j0"
server_key: "7471facee7558a8bdf33060c2ffb4f55"
server_key_iv: "efae62415533def68454351cf6d437da"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:54.716286 13088 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 03:01:54.733791 13103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:54.735816 13088 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 03:01:54.736012 13088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "2baa988c223b4c6aad408da9fd0c92b0"
format_stamp: "Formatted at 2026-06-20 03:01:54 on dist-test-slave-f1j0"
server_key: "7471facee7558a8bdf33060c2ffb4f55"
server_key_iv: "efae62415533def68454351cf6d437da"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:54.736274 13088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:01:54.760327 13088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:01:54.764453 13088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:01:54.764706 13088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:01:54.765614 13088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:01:54.766893 13088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:01:54.766960 13088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:54.767041 13088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:01:54.767083 13088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:54.799304 13088 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34069
I20260620 03:01:54.799392 13215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34069 every 8 connection(s)
I20260620 03:01:54.800654 13088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:01:54.810685 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 13088
I20260620 03:01:54.810871 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:01:54.811307 388 external_mini_cluster.cc:1428] Setting key 5e5bd0e4cd7fa0a1f5192c2605d1657f
Jun 20 03:01:54 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:01:54.819577 13218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:54.803214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57379 (local address 127.0.97.62:39679)
0620 03:01:54.803679 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:54.803684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:54.804634 (+ 950us) server_negotiation.cc:407] Connection header received
0620 03:01:54.806444 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:54.806449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:54.806548 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:54.806664 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:54.807991 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:54.808701 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:54.810264 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:54.810698 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:54.810837 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:54.814532 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:54.814539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:54.814544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:54.814583 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:54.816991 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:54.817682 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:54.817686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:54.817688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:54.817842 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:54.818205 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:54.818208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:54.818210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:54.818467 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:54.818570 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:54.819172 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 03:01:54.819341 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":212,"threads_started":1}
I20260620 03:01:54.820608 13219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:54.803736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39679 (local address 127.0.97.3:57379)
0620 03:01:54.804477 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 03:01:54.804514 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 03:01:54.806173 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:01:54.806892 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:01:54.806900 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:01:54.807294 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:01:54.807830 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:54.807847 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:54.808844 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:54.808849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:54.810128 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:54.810139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:54.810938 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:54.810947 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:54.811178 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:54.811890 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:01:54.811940 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:01:54.814303 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:01:54.817107 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:01:54.817113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:01:54.817127 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:01:54.817523 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:01:54.817932 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:01:54.817936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:01:54.817938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:01:54.818109 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:01:54.818566 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:01:54.818571 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:01:54.818864 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 03:01:54.819092 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 03:01:54.819443 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":329,"threads_started":1}
I20260620 03:01:54.821992 13216 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39679
I20260620 03:01:54.822309 13216 heartbeater.cc:464] Registering TS with master...
I20260620 03:01:54.822964 13216 heartbeater.cc:511] Master 127.0.97.62:39679 requested a full tablet report, sending...
I20260620 03:01:54.824273 12726 ts_manager.cc:194] Registered new tserver with Master: 2baa988c223b4c6aad408da9fd0c92b0 (127.0.97.3:34069)
I20260620 03:01:54.825067 12726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57379
I20260620 03:01:54.827005 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:01:54.828704 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 12806
Jun 20 03:01:54 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:01:54.845410 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:54.829761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.0.97.1:39043)
0620 03:01:54.830358 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:54.830364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:54.830388 (+ 24us) server_negotiation.cc:407] Connection header received
0620 03:01:54.830447 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:54.830453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:54.830642 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:54.830774 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:54.832454 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:54.833551 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:54.835061 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:54.835598 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:54.835693 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:54.840094 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:54.840106 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:54.840121 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:54.840154 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:54.842874 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:54.843494 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:54.843501 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:54.843503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:54.843643 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:54.844025 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:54.844028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:54.844030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:54.844385 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:54.844602 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:54.844912 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 03:01:54.845093 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":306,"threads_started":1}
I20260620 03:01:54.846660 12934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:01:54.928380 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 12806
I20260620 03:01:54.944861 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 12945
Jun 20 03:01:54 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:01:54.960242 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:54.945837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.0.97.2:37789)
0620 03:01:54.946353 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:54.946361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:54.946386 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:01:54.946444 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:54.946450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:54.946668 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:54.946789 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:54.947945 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:54.948929 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:54.950454 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:54.951011 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:54.951113 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:54.955268 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:54.955279 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:54.955293 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:54.955324 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:54.957856 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:54.958484 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:54.958491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:54.958493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:54.958610 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:54.958899 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:54.958902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:54.958904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:54.959211 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:54.959420 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:54.959668 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 03:01:54.959866 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":281,"threads_started":1}
I20260620 03:01:54.961480 13077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:01:55.042778 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 12945
I20260620 03:01:55.058435 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 13088
Jun 20 03:01:55 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:01:55.073457 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:55.059383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53864 (local address 127.0.97.3:34069)
0620 03:01:55.059968 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:55.059975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:55.060002 (+ 27us) server_negotiation.cc:407] Connection header received
0620 03:01:55.060058 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:55.060064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:55.060271 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:55.060406 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:55.061655 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:55.062630 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:55.063816 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:55.064350 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:55.064437 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:55.068493 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:55.068502 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:55.068517 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:55.068545 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:55.071120 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:55.071736 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:55.071742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:55.071745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:55.071858 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:55.072147 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:55.072150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:55.072152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:55.072448 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:55.072670 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:55.072903 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 03:01:55.073070 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":296,"threads_started":1}
I20260620 03:01:55.074774 13211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:01:55.157886 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 13088
I20260620 03:01:55.173722 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 12694
I20260620 03:01:55.175027 12756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:01:55.281548 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 12694
I20260620 03:01:55.305081 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39679
--webserver_interface=127.0.97.62
--webserver_port=38273
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39679
--ranger_config_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41007/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41007/kms with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:01:55.629029 13228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:01:55.629372 13228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:01:55.629465 13228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:01:55.636816 13228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:01:55.636922 13228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:01:55.636968 13228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:01:55.637023 13228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:01:55.637069 13228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:01:55.647194 13228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41007/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39679
--ranger_config_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:01:55.648435 13228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:01:55.649739 13228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:01:55.665053 13236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:55.665000 13233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:55.665057 13234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:01:55.665953 13228 server_base.cc:1034] running on GCE node
I20260620 03:01:55.666759 13228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:01:55.668092 13228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:01:55.669333 13228 hybrid_clock.cc:648] HybridClock initialized: now 1781924515669293 us; error 57 us; skew 500 ppm
I20260620 03:01:55.669735 13228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:01:55 dist-test-slave-f1j0 krb5kdc[12250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:01:55.675477 13228 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:01:55.676961 13228 webserver.cc:460] Webserver started at http://127.0.97.62:38273/ using document root <none> and password file <none>
I20260620 03:01:55.677697 13228 fs_manager.cc:359] Metadata directory not provided
I20260620 03:01:55.677814 13228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 03:01:55 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:01:55.728457 13228 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:01:55.732133 13244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:55.733928 13228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:01:55.734122 13228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "2f464c6ca77e4a298dd81f9a78794dff"
format_stamp: "Formatted at 2026-06-20 03:01:51 on dist-test-slave-f1j0"
server_key: "242a53f2419a934d925e769c1524cbf8"
server_key_iv: "19714181e648f046f367cb874705ae53"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:55.735477 13228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:01:55.752890 13228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:01:55.756543 13228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:01:55.756780 13228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:01:55.775173 13228 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39679
I20260620 03:01:55.775197 13295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39679 every 8 connection(s)
I20260620 03:01:55.776719 13228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:01:55.783507 13296 sys_catalog.cc:263] Verifying existing consensus state
I20260620 03:01:55.786828 13296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff: Bootstrap starting.
I20260620 03:01:55.787299 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 13228
I20260620 03:01:55.788164 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39043
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39679
--webserver_port=43849
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41007/kms with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:01:55.805730 13296 log.cc:824] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff: Log is configured to *not* fsync() on all Append() calls
I20260620 03:01:55.814854 13296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 03:01:55.815393 13296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff: Bootstrap complete.
I20260620 03:01:55.824245 13296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f464c6ca77e4a298dd81f9a78794dff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39679 } }
I20260620 03:01:55.825116 13296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f464c6ca77e4a298dd81f9a78794dff, State: Initialized, Role: FOLLOWER
I20260620 03:01:55.825754 13296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f464c6ca77e4a298dd81f9a78794dff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39679 } }
I20260620 03:01:55.825994 13296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:01:55.826118 13296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:01:55.826296 13296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [term 1 FOLLOWER]: Advancing to term 2
I20260620 03:01:55.830034 13296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f464c6ca77e4a298dd81f9a78794dff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39679 } }
I20260620 03:01:55.830509 13296 leader_election.cc:302] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f464c6ca77e4a298dd81f9a78794dff; no voters:
I20260620 03:01:55.831213 13296 leader_election.cc:288] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 03:01:55.831357 13300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [term 2 FOLLOWER]: Leader election won for term 2
I20260620 03:01:55.833617 13296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:01:55.833845 13300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [term 2 LEADER]: Becoming Leader. State: Replica: 2f464c6ca77e4a298dd81f9a78794dff, State: Running, Role: LEADER
I20260620 03:01:55.834455 13300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f464c6ca77e4a298dd81f9a78794dff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39679 } }
I20260620 03:01:55.844823 13301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2f464c6ca77e4a298dd81f9a78794dff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f464c6ca77e4a298dd81f9a78794dff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39679 } } }
I20260620 03:01:55.845292 13301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [sys.catalog]: This master's current role is: LEADER
I20260620 03:01:55.847281 13302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f464c6ca77e4a298dd81f9a78794dff. Latest consensus state: current_term: 2 leader_uuid: "2f464c6ca77e4a298dd81f9a78794dff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f464c6ca77e4a298dd81f9a78794dff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39679 } } }
I20260620 03:01:55.847622 13302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff [sys.catalog]: This master's current role is: LEADER
I20260620 03:01:55.847689 13311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:01:55.853641 13311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:01:55.854372 13296 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 03:01:55.855288 13311 catalog_manager.cc:1259] Loaded cluster ID: 372755a267174c9da5a3e14590734e99
I20260620 03:01:55.855382 13311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:01:55.859431 13311 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:01:55.860977 13311 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff: Loaded TSK: 0
I20260620 03:01:55.862663 13311 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 03:01:56.256057 13298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:01:56.256592 13298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:01:56.256881 13298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:01:56.268713 13298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:01:56.268926 13298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:01:56.269176 13298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:01:56.285223 13298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43849
--tserver_master_addrs=127.0.97.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:01:56.287132 13298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:01:56.288992 13298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:01:56.318742 13335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:56.320197 13334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:56.322033 13337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:01:56.323469 13298 server_base.cc:1034] running on GCE node
I20260620 03:01:56.324251 13298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:01:56.325353 13298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:01:56.326663 13298 hybrid_clock.cc:648] HybridClock initialized: now 1781924516326567 us; error 77 us; skew 500 ppm
I20260620 03:01:56.327025 13298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:01:56 dist-test-slave-f1j0 krb5kdc[12250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:01:56.330504 13298 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:01:56.332525 13298 webserver.cc:460] Webserver started at http://127.0.97.1:43849/ using document root <none> and password file <none>
I20260620 03:01:56.333280 13298 fs_manager.cc:359] Metadata directory not provided
I20260620 03:01:56.333417 13298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 03:01:56 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:01:56.358904 13298 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 03:01:56.369977 13345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:56.374555 13298 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.010s sys 0.001s
I20260620 03:01:56.374801 13298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "6ee2b5240a8b4ec0a65bfc4bf25d17ef"
format_stamp: "Formatted at 2026-06-20 03:01:53 on dist-test-slave-f1j0"
server_key: "8cf6b4af28de1256a56db94910bd1e92"
server_key_iv: "94048e52a802f2e6c3f84d07d6aaa097"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:56.376300 13298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:01:56.435904 13298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:01:56.441339 13298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:01:56.441720 13298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:01:56.442831 13298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:01:56.444373 13298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:01:56.444463 13298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:56.444588 13298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:01:56.444669 13298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:56.505232 13298 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39043
I20260620 03:01:56.505390 13458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39043 every 8 connection(s)
I20260620 03:01:56.507247 13298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:01:56.509861 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 13298
I20260620 03:01:56.510707 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37789
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39679
--webserver_port=36415
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41007/kms with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:01:56 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:01:56.542493 13462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:56.513822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42087 (local address 127.0.97.62:39679)
0620 03:01:56.519481 (+ 5659us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:56.519502 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:56.519553 (+ 51us) server_negotiation.cc:407] Connection header received
0620 03:01:56.519992 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:56.520014 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:56.520513 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:56.520928 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:56.522976 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:56.524924 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:56.527509 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:56.528532 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:56.529640 (+ 1108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:56.533825 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:56.533836 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:56.533859 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:56.533912 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:56.537202 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:56.538082 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:56.538088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:56.538095 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:56.538254 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:56.538750 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:56.538760 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:56.538763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:56.539383 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:56.539656 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:56.540594 (+ 938us) server_negotiation.cc:299] Negotiation successful
0620 03:01:56.540991 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5372,"thread_start_us":5164,"threads_started":1}
I20260620 03:01:56.542505 13461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:56.511033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39679 (local address 127.0.97.1:42087)
0620 03:01:56.511945 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0620 03:01:56.512004 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 03:01:56.514957 (+ 2953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:01:56.521150 (+ 6193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:01:56.521173 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:01:56.521879 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:01:56.522690 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:56.522718 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:56.525131 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:56.525138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:56.527254 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:56.527279 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:56.528809 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:56.528816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:56.529093 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:56.529926 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:01:56.529961 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:01:56.532649 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:01:56.537362 (+ 4713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:01:56.537371 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:01:56.537389 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:01:56.537919 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:01:56.538376 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:01:56.538381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:01:56.538385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:01:56.538609 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:01:56.539583 (+ 974us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:01:56.539591 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:01:56.540075 (+ 484us) client_negotiation.cc:769] Sending connection context
0620 03:01:56.540351 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 03:01:56.541036 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":1248,"threads_started":1}
I20260620 03:01:56.546061 13459 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39679
I20260620 03:01:56.546442 13459 heartbeater.cc:464] Registering TS with master...
I20260620 03:01:56.547161 13459 heartbeater.cc:511] Master 127.0.97.62:39679 requested a full tablet report, sending...
W20260620 03:01:56.550280 13459 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39679 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 03:01:56.550521 13459 heartbeater.cc:464] Registering TS with master...
I20260620 03:01:56.550817 13459 heartbeater.cc:511] Master 127.0.97.62:39679 requested a full tablet report, sending...
I20260620 03:01:56.551925 13459 heartbeater.cc:464] Registering TS with master...
I20260620 03:01:56.552178 13459 heartbeater.cc:511] Master 127.0.97.62:39679 requested a full tablet report, sending...
W20260620 03:01:56.553237 13459 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 03:01:56.972204 13463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:01:56.972497 13463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:01:56.972645 13463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:01:56.982421 13463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:01:56.982576 13463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:01:56.982777 13463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:01:56.998667 13463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36415
--tserver_master_addrs=127.0.97.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:01:57.000546 13463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:01:57.002393 13463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:01:57.018605 13477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:57.018730 13480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:57.019461 13478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:01:57.020678 13463 server_base.cc:1034] running on GCE node
I20260620 03:01:57.021369 13463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:01:57.022558 13463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:01:57.023818 13463 hybrid_clock.cc:648] HybridClock initialized: now 1781924517023743 us; error 55 us; skew 500 ppm
I20260620 03:01:57.024171 13463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:01:57 dist-test-slave-f1j0 krb5kdc[12250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:01:57.028266 13463 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:01:57.030160 13463 webserver.cc:460] Webserver started at http://127.0.97.2:36415/ using document root <none> and password file <none>
I20260620 03:01:57.031113 13463 fs_manager.cc:359] Metadata directory not provided
I20260620 03:01:57.031286 13463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 03:01:57 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:01:57.075429 13463 fs_manager.cc:709] Time spent opening directory manager: real 0.024s user 0.004s sys 0.011s
I20260620 03:01:57.086429 13488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:57.088155 13463 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.011s sys 0.000s
I20260620 03:01:57.088374 13463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "28d3927fd978440dbea6a4d20ef45851"
format_stamp: "Formatted at 2026-06-20 03:01:54 on dist-test-slave-f1j0"
server_key: "a92f6ad6f64672fdc13712c76e65f242"
server_key_iv: "0f7d354bb90df99968e277c5632c7b41"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:57.089497 13463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:01:57.135295 13463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:01:57.140749 13463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:01:57.141042 13463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:01:57.142238 13463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:01:57.144068 13463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:01:57.144179 13463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:57.144335 13463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:01:57.144434 13463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 03:01:57 dist-test-slave-f1j0 krb5kdc[12250](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781924517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:01:57.209303 13463 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37789
I20260620 03:01:57.209368 13602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37789 every 8 connection(s)
I20260620 03:01:57.211333 13463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:01:57.220240 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 13463
I20260620 03:01:57.221148 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34069
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39679
--webserver_port=43213
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41007/kms with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:01:57 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:01:57.267620 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:57.215681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45935 (local address 127.0.97.62:39679)
0620 03:01:57.237029 (+ 21348us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:57.237037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:57.239280 (+ 2243us) server_negotiation.cc:407] Connection header received
0620 03:01:57.242138 (+ 2858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:57.242145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:57.242239 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:57.242322 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:57.249594 (+ 7272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:57.250739 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:57.253246 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:57.253961 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:57.254058 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:57.258802 (+ 4744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:57.258809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:57.258815 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:57.258857 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:57.262925 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:57.264046 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:57.264053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:57.264057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:57.264215 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:57.264756 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:57.264761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:57.264764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:57.265139 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:57.265292 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:57.266949 (+ 1657us) server_negotiation.cc:299] Negotiation successful
0620 03:01:57.267185 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21194,"thread_start_us":21361,"threads_started":1}
I20260620 03:01:57.269477 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:57.215028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39679 (local address 127.0.97.2:45935)
0620 03:01:57.237810 (+ 22782us) negotiation.cc:107] Waiting for socket to connect
0620 03:01:57.237880 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 03:01:57.241770 (+ 3890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:01:57.242920 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:01:57.242933 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:01:57.243637 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:01:57.244487 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:57.244511 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:57.250954 (+ 6443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:57.250961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:57.253052 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:57.253078 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:57.254437 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:57.254444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:57.254750 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:57.255585 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:01:57.255615 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:01:57.258489 (+ 2874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:01:57.263101 (+ 4612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:01:57.263113 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:01:57.263138 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:01:57.263872 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:01:57.264352 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:01:57.264357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:01:57.264362 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:01:57.264599 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:01:57.265284 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:01:57.265292 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:01:57.265748 (+ 456us) client_negotiation.cc:769] Sending connection context
0620 03:01:57.266043 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 03:01:57.266618 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22386,"thread_start_us":25209,"threads_started":1}
I20260620 03:01:57.271341 13603 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39679
I20260620 03:01:57.271659 13603 heartbeater.cc:464] Registering TS with master...
I20260620 03:01:57.272475 13603 heartbeater.cc:511] Master 127.0.97.62:39679 requested a full tablet report, sending...
I20260620 03:01:57.274565 13261 ts_manager.cc:194] Registered new tserver with Master: 28d3927fd978440dbea6a4d20ef45851 (127.0.97.2:37789)
I20260620 03:01:57.276790 13261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45935
I20260620 03:01:57.553390 13459 heartbeater.cc:464] Registering TS with master...
I20260620 03:01:57.553890 13459 heartbeater.cc:511] Master 127.0.97.62:39679 requested a full tablet report, sending...
I20260620 03:01:57.555227 13261 ts_manager.cc:194] Registered new tserver with Master: 6ee2b5240a8b4ec0a65bfc4bf25d17ef (127.0.97.1:39043)
I20260620 03:01:57.556401 13261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42087
Jun 20 03:01:57 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 03:01:57.680711 13607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:01:57.681212 13607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:01:57.681412 13607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:01:57.693456 13607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:01:57.693763 13607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:01:57.693989 13607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:01:57.711166 13607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43213
--tserver_master_addrs=127.0.97.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:01:57.713085 13607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:01:57.714998 13607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:01:57.729990 13613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:57.730062 13614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:01:57.730036 13616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:01:57.731066 13607 server_base.cc:1034] running on GCE node
I20260620 03:01:57.731868 13607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:01:57.733182 13607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:01:57.734423 13607 hybrid_clock.cc:648] HybridClock initialized: now 1781924517734343 us; error 60 us; skew 500 ppm
I20260620 03:01:57.734898 13607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:01:57 dist-test-slave-f1j0 krb5kdc[12250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:01:57.738411 13607 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:01:57.740415 13607 webserver.cc:460] Webserver started at http://127.0.97.3:43213/ using document root <none> and password file <none>
I20260620 03:01:57.741137 13607 fs_manager.cc:359] Metadata directory not provided
I20260620 03:01:57.741310 13607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 03:01:57 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:01:57.775832 13607 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.004s sys 0.004s
I20260620 03:01:57.790282 13624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:57.792093 13607 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.001s sys 0.012s
I20260620 03:01:57.792294 13607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "2baa988c223b4c6aad408da9fd0c92b0"
format_stamp: "Formatted at 2026-06-20 03:01:54 on dist-test-slave-f1j0"
server_key: "7471facee7558a8bdf33060c2ffb4f55"
server_key_iv: "efae62415533def68454351cf6d437da"
server_key_version: "kuduclusterkey@0"
I20260620 03:01:57.793440 13607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:01:57.815069 13607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:01:57.819315 13607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:01:57.819597 13607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:01:57.820632 13607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:01:57.822072 13607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:01:57.822191 13607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:57.822329 13607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:01:57.822413 13607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:01:57.861557 13607 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34069
I20260620 03:01:57.861667 13736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34069 every 8 connection(s)
I20260620 03:01:57.863598 13607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:01:57.869096 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 13607
Jun 20 03:01:57 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:01:57.917881 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:57.873350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39679 (local address 127.0.97.3:39673)
0620 03:01:57.883023 (+ 9673us) negotiation.cc:107] Waiting for socket to connect
0620 03:01:57.883090 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 03:01:57.888581 (+ 5491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:01:57.894727 (+ 6146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:01:57.894742 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:01:57.895370 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:01:57.896108 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:57.896126 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:57.897801 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:57.897807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:57.899498 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:57.899518 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:57.901628 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:57.901637 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:57.901969 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:57.904803 (+ 2834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:01:57.904834 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:01:57.907661 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:01:57.912904 (+ 5243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:01:57.912913 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:01:57.912932 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:01:57.913533 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:01:57.914010 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:01:57.914015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:01:57.914019 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:01:57.914274 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:01:57.915007 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:01:57.915015 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:01:57.915468 (+ 453us) client_negotiation.cc:769] Sending connection context
0620 03:01:57.915804 (+ 336us) client_negotiation.cc:240] Negotiation successful
0620 03:01:57.916388 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9221,"thread_start_us":12084,"threads_started":1}
I20260620 03:01:57.919713 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:57.893170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39673 (local address 127.0.97.62:39679)
0620 03:01:57.894072 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:57.894078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:57.894097 (+ 19us) server_negotiation.cc:407] Connection header received
0620 03:01:57.894145 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:57.894150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:57.894219 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:57.894328 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:57.896335 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:57.897538 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:57.899654 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:57.901177 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:57.901305 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:57.907986 (+ 6681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:57.907994 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:57.908002 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:57.908062 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:57.912748 (+ 4686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:57.913711 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:57.913716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:57.913720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:57.913872 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:57.914407 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:57.914412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:57.914416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:57.914794 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:57.914925 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:57.919117 (+ 4192us) server_negotiation.cc:299] Negotiation successful
0620 03:01:57.919315 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":783,"thread_start_us":643,"threads_started":1}
I20260620 03:01:57.920055 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:57.870154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34252 (local address 127.0.97.62:39679)
0620 03:01:57.893188 (+ 23034us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:57.893194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:57.893218 (+ 24us) server_negotiation.cc:407] Connection header received
0620 03:01:57.893286 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:57.893291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:57.893393 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:57.893520 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:57.895630 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:57.896807 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:57.899576 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:57.900416 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:57.900543 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:57.904383 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:57.904390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:57.904395 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:57.904445 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:57.908914 (+ 4469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:57.912204 (+ 3290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:57.912210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:57.912213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:57.912371 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:57.918494 (+ 6123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:57.918497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:57.918500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:57.918801 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:57.919014 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:57.919446 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 03:01:57.919665 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":22901,"thread_start_us":22788,"threads_started":1}
I20260620 03:01:57.921542 13737 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39679
I20260620 03:01:57.921907 13737 heartbeater.cc:464] Registering TS with master...
I20260620 03:01:57.922863 13737 heartbeater.cc:511] Master 127.0.97.62:39679 requested a full tablet report, sending...
I20260620 03:01:57.925503 13261 ts_manager.cc:194] Registered new tserver with Master: 2baa988c223b4c6aad408da9fd0c92b0 (127.0.97.3:34069)
I20260620 03:01:57.926640 13261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39673
I20260620 03:01:57.939284 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 03:01:57 dist-test-slave-f1j0 krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 03:01:57.971415 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:57.942760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34262 (local address 127.0.97.62:39679)
0620 03:01:57.942947 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:57.942953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:57.954248 (+ 11295us) server_negotiation.cc:407] Connection header received
0620 03:01:57.954465 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:57.954471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:57.954566 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:57.954662 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:57.956816 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:57.957987 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:57.959961 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:57.960609 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:57.960722 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:57.964367 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:57.964375 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:57.964380 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:57.964424 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:57.968019 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:57.968880 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:57.968894 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:57.968898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:57.969045 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:57.969980 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:57.969985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:57.969989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:57.970361 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:57.970629 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:57.970946 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 03:01:57.971111 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 03:01:57.977670 13261 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:01:57.989444 13261 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:01:58.038625 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:58.024720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39043 (local address 127.0.0.1:56698)
0620 03:01:58.026466 (+ 1746us) negotiation.cc:107] Waiting for socket to connect
0620 03:01:58.026526 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 03:01:58.026746 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:01:58.028110 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:01:58.028117 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:01:58.028150 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:01:58.028536 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:58.028557 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.030577 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.030583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:58.034147 (+ 3564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:58.034165 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.037555 (+ 3390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.037562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:58.037840 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:58.037900 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 03:01:58.038041 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 03:01:58.038246 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1367,"thread_start_us":1191,"threads_started":1}
I20260620 03:01:58.040210 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:58.025004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56698 (local address 127.0.97.1:39043)
0620 03:01:58.027353 (+ 2349us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:58.027363 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:58.027392 (+ 29us) server_negotiation.cc:407] Connection header received
0620 03:01:58.027492 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:58.027499 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:58.027764 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:58.027952 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:01:58.028765 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.030423 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.034430 (+ 4007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.037042 (+ 2612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.037168 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:58.037411 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:58.039701 (+ 2290us) server_negotiation.cc:299] Negotiation successful
0620 03:01:58.039875 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2153,"thread_start_us":2841,"threads_started":1}
I20260620 03:01:58.044896 13394 tablet_service.cc:1467] Processing CreateTablet for tablet b572767e7c8344c5a4f49d9748dc17c1 (DEFAULT_TABLE table=test-table [id=d470c3a24159480b997058e81d8b6184]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:01:58.046633 13394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b572767e7c8344c5a4f49d9748dc17c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:01:58.068850 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:58.027545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37789 (local address 127.0.0.1:53320)
0620 03:01:58.039515 (+ 11970us) negotiation.cc:107] Waiting for socket to connect
0620 03:01:58.039536 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:01:58.039648 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:01:58.061936 (+ 22288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:01:58.061942 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:01:58.061977 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:01:58.062207 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:58.062221 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.064407 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.064413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:58.066334 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:58.066350 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.068100 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.068107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:58.068294 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:58.068331 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 03:01:58.068400 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 03:01:58.068538 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11849,"thread_start_us":36615,"threads_started":1}
I20260620 03:01:58.070355 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:58.028095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.0.97.2:37789)
0620 03:01:58.061172 (+ 33077us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:58.061182 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:58.061213 (+ 31us) server_negotiation.cc:407] Connection header received
0620 03:01:58.061352 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:58.061361 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:58.061677 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:58.062417 (+ 740us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:01:58.062652 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.064245 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.066524 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.067924 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.069535 (+ 1611us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:58.069785 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:58.069850 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 03:01:58.070009 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32900,"thread_start_us":33547,"threads_started":1}
I20260620 03:01:58.074383 13538 tablet_service.cc:1467] Processing CreateTablet for tablet b572767e7c8344c5a4f49d9748dc17c1 (DEFAULT_TABLE table=test-table [id=d470c3a24159480b997058e81d8b6184]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:01:58.075893 13538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b572767e7c8344c5a4f49d9748dc17c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:01:58.078609 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:58.041178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34069 (local address 127.0.0.1:53880)
0620 03:01:58.062201 (+ 21023us) negotiation.cc:107] Waiting for socket to connect
0620 03:01:58.062223 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:01:58.062353 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:01:58.071115 (+ 8762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:01:58.071120 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:01:58.071161 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:01:58.071531 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:58.071545 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.073873 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.073879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:58.075936 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:58.075952 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.077845 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.077851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:58.078020 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:58.078058 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 03:01:58.078129 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 03:01:58.078283 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20886}
I20260620 03:01:58.079988 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:58.041831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53880 (local address 127.0.97.3:34069)
0620 03:01:58.061535 (+ 19704us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:58.061541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:58.070501 (+ 8960us) server_negotiation.cc:407] Connection header received
0620 03:01:58.070617 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:58.070626 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:58.070891 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:58.071758 (+ 867us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:01:58.072067 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.073700 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.076133 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.077608 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.079186 (+ 1578us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:58.079412 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:58.079482 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 03:01:58.079641 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19556,"thread_start_us":19781,"threads_started":1}
I20260620 03:01:58.081933 13758 tablet_bootstrap.cc:492] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef: Bootstrap starting.
I20260620 03:01:58.085672 13672 tablet_service.cc:1467] Processing CreateTablet for tablet b572767e7c8344c5a4f49d9748dc17c1 (DEFAULT_TABLE table=test-table [id=d470c3a24159480b997058e81d8b6184]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:01:58.087271 13672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b572767e7c8344c5a4f49d9748dc17c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:01:58.087996 13758 tablet_bootstrap.cc:654] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef: Neither blocks nor log segments found. Creating new log.
I20260620 03:01:58.089401 13758 log.cc:824] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef: Log is configured to *not* fsync() on all Append() calls
I20260620 03:01:58.099697 13760 tablet_bootstrap.cc:492] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851: Bootstrap starting.
I20260620 03:01:58.103768 13760 tablet_bootstrap.cc:654] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851: Neither blocks nor log segments found. Creating new log.
I20260620 03:01:58.105237 13760 log.cc:824] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851: Log is configured to *not* fsync() on all Append() calls
I20260620 03:01:58.114900 13760 tablet_bootstrap.cc:492] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851: No bootstrap required, opened a new log
I20260620 03:01:58.115309 13760 ts_tablet_manager.cc:1397] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
I20260620 03:01:58.118367 13758 tablet_bootstrap.cc:492] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef: No bootstrap required, opened a new log
I20260620 03:01:58.118770 13758 ts_tablet_manager.cc:1397] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef: Time spent bootstrapping tablet: real 0.037s user 0.019s sys 0.014s
I20260620 03:01:58.128832 13760 raft_consensus.cc:348] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee2b5240a8b4ec0a65bfc4bf25d17ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } } peers { permanent_uuid: "28d3927fd978440dbea6a4d20ef45851" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } } peers { permanent_uuid: "2baa988c223b4c6aad408da9fd0c92b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34069 } }
I20260620 03:01:58.129535 13760 raft_consensus.cc:374] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:01:58.129323 13758 raft_consensus.cc:348] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee2b5240a8b4ec0a65bfc4bf25d17ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } } peers { permanent_uuid: "28d3927fd978440dbea6a4d20ef45851" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } } peers { permanent_uuid: "2baa988c223b4c6aad408da9fd0c92b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34069 } }
I20260620 03:01:58.129716 13760 raft_consensus.cc:729] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28d3927fd978440dbea6a4d20ef45851, State: Initialized, Role: FOLLOWER
I20260620 03:01:58.129912 13758 raft_consensus.cc:374] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:01:58.130067 13758 raft_consensus.cc:729] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ee2b5240a8b4ec0a65bfc4bf25d17ef, State: Initialized, Role: FOLLOWER
I20260620 03:01:58.130523 13760 consensus_queue.cc:260] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee2b5240a8b4ec0a65bfc4bf25d17ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } } peers { permanent_uuid: "28d3927fd978440dbea6a4d20ef45851" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } } peers { permanent_uuid: "2baa988c223b4c6aad408da9fd0c92b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34069 } }
I20260620 03:01:58.130847 13758 consensus_queue.cc:260] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee2b5240a8b4ec0a65bfc4bf25d17ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } } peers { permanent_uuid: "28d3927fd978440dbea6a4d20ef45851" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } } peers { permanent_uuid: "2baa988c223b4c6aad408da9fd0c92b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34069 } }
I20260620 03:01:58.137647 13764 tablet_bootstrap.cc:492] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0: Bootstrap starting.
I20260620 03:01:58.138811 13603 heartbeater.cc:503] Master 127.0.97.62:39679 was elected leader, sending a full tablet report...
I20260620 03:01:58.139947 13760 ts_tablet_manager.cc:1428] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851: Time spent starting tablet: real 0.025s user 0.012s sys 0.008s
I20260620 03:01:58.142231 13758 ts_tablet_manager.cc:1428] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef: Time spent starting tablet: real 0.023s user 0.017s sys 0.002s
I20260620 03:01:58.142691 13459 heartbeater.cc:503] Master 127.0.97.62:39679 was elected leader, sending a full tablet report...
I20260620 03:01:58.144160 13764 tablet_bootstrap.cc:654] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0: Neither blocks nor log segments found. Creating new log.
I20260620 03:01:58.145702 13764 log.cc:824] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0: Log is configured to *not* fsync() on all Append() calls
I20260620 03:01:58.166074 13764 tablet_bootstrap.cc:492] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0: No bootstrap required, opened a new log
I20260620 03:01:58.166504 13764 ts_tablet_manager.cc:1397] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.013s
I20260620 03:01:58.178856 13764 raft_consensus.cc:348] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee2b5240a8b4ec0a65bfc4bf25d17ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } } peers { permanent_uuid: "28d3927fd978440dbea6a4d20ef45851" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } } peers { permanent_uuid: "2baa988c223b4c6aad408da9fd0c92b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34069 } }
I20260620 03:01:58.179369 13764 raft_consensus.cc:374] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:01:58.179474 13764 raft_consensus.cc:729] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2baa988c223b4c6aad408da9fd0c92b0, State: Initialized, Role: FOLLOWER
I20260620 03:01:58.180171 13764 consensus_queue.cc:260] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee2b5240a8b4ec0a65bfc4bf25d17ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } } peers { permanent_uuid: "28d3927fd978440dbea6a4d20ef45851" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } } peers { permanent_uuid: "2baa988c223b4c6aad408da9fd0c92b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34069 } }
I20260620 03:01:58.181357 13737 heartbeater.cc:503] Master 127.0.97.62:39679 was elected leader, sending a full tablet report...
I20260620 03:01:58.182128 13764 ts_tablet_manager.cc:1428] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260620 03:01:58.211722 13766 raft_consensus.cc:482] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:01:58.212028 13766 raft_consensus.cc:504] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee2b5240a8b4ec0a65bfc4bf25d17ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } } peers { permanent_uuid: "28d3927fd978440dbea6a4d20ef45851" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } } peers { permanent_uuid: "2baa988c223b4c6aad408da9fd0c92b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34069 } }
I20260620 03:01:58.214397 13766 leader_election.cc:288] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ee2b5240a8b4ec0a65bfc4bf25d17ef (127.0.97.1:39043), 28d3927fd978440dbea6a4d20ef45851 (127.0.97.2:37789)
I20260620 03:01:58.222412 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:58.214881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39043 (local address 127.0.97.3:54863)
0620 03:01:58.215397 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 03:01:58.215416 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:01:58.215524 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:01:58.216475 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:01:58.216480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:01:58.216509 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:01:58.216701 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:58.216710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.218147 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.218152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:58.220084 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:58.220100 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.221670 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.221676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:58.221867 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:58.221906 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 03:01:58.221988 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 03:01:58.222123 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396}
I20260620 03:01:58.223212 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:58.215742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54863 (local address 127.0.97.1:39043)
0620 03:01:58.215998 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:58.216007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:58.216034 (+ 27us) server_negotiation.cc:407] Connection header received
0620 03:01:58.216127 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:58.216135 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:58.216252 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:58.216370 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:01:58.216865 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.217959 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.220280 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.221449 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.222680 (+ 1231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:58.222780 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:58.222828 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 03:01:58.222967 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 03:01:58.224093 13414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b572767e7c8344c5a4f49d9748dc17c1" candidate_uuid: "2baa988c223b4c6aad408da9fd0c92b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ee2b5240a8b4ec0a65bfc4bf25d17ef" is_pre_election: true
I20260620 03:01:58.224695 13414 raft_consensus.cc:2454] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2baa988c223b4c6aad408da9fd0c92b0 in term 0.
I20260620 03:01:58.225541 13625 leader_election.cc:302] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2baa988c223b4c6aad408da9fd0c92b0, 6ee2b5240a8b4ec0a65bfc4bf25d17ef; no voters:
I20260620 03:01:58.233378 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:58.215922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53855 (local address 127.0.97.2:37789)
0620 03:01:58.216702 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:58.216711 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:58.226198 (+ 9487us) server_negotiation.cc:407] Connection header received
0620 03:01:58.226571 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:58.226577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:58.226677 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:58.226751 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:01:58.227409 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.228554 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.230888 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.232164 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.232283 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:58.232383 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:58.232960 (+ 577us) server_negotiation.cc:299] Negotiation successful
0620 03:01:58.233097 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605}
I20260620 03:01:58.234098 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:58.214928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37789 (local address 127.0.97.3:53855)
0620 03:01:58.226088 (+ 11160us) negotiation.cc:107] Waiting for socket to connect
0620 03:01:58.226107 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:01:58.226305 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:01:58.227012 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:01:58.227016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:01:58.227043 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:01:58.227241 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:58.227253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.228874 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.228878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:58.230684 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:01:58.230699 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.232539 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.232546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:01:58.232767 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:58.232802 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 03:01:58.233648 (+ 846us) client_negotiation.cc:240] Negotiation successful
0620 03:01:58.233795 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11060,"threads_started":1}
I20260620 03:01:58.234349 13766 raft_consensus.cc:2781] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:01:58.234529 13766 raft_consensus.cc:482] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:01:58.234639 13766 raft_consensus.cc:3037] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:01:58.238062 13766 raft_consensus.cc:504] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee2b5240a8b4ec0a65bfc4bf25d17ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } } peers { permanent_uuid: "28d3927fd978440dbea6a4d20ef45851" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } } peers { permanent_uuid: "2baa988c223b4c6aad408da9fd0c92b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34069 } }
I20260620 03:01:58.239557 13414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b572767e7c8344c5a4f49d9748dc17c1" candidate_uuid: "2baa988c223b4c6aad408da9fd0c92b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ee2b5240a8b4ec0a65bfc4bf25d17ef"
I20260620 03:01:58.239813 13414 raft_consensus.cc:3037] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:01:58.243126 13414 raft_consensus.cc:2454] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2baa988c223b4c6aad408da9fd0c92b0 in term 1.
I20260620 03:01:58.243824 13625 leader_election.cc:302] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2baa988c223b4c6aad408da9fd0c92b0, 6ee2b5240a8b4ec0a65bfc4bf25d17ef; no voters:
I20260620 03:01:58.245532 13766 leader_election.cc:288] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ee2b5240a8b4ec0a65bfc4bf25d17ef (127.0.97.1:39043), 28d3927fd978440dbea6a4d20ef45851 (127.0.97.2:37789)
I20260620 03:01:58.246014 13768 raft_consensus.cc:2781] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:01:58.246810 13768 raft_consensus.cc:686] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [term 1 LEADER]: Becoming Leader. State: Replica: 2baa988c223b4c6aad408da9fd0c92b0, State: Running, Role: LEADER
I20260620 03:01:58.246821 13557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b572767e7c8344c5a4f49d9748dc17c1" candidate_uuid: "2baa988c223b4c6aad408da9fd0c92b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28d3927fd978440dbea6a4d20ef45851"
I20260620 03:01:58.247296 13557 raft_consensus.cc:3037] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:01:58.247630 13768 consensus_queue.cc:237] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee2b5240a8b4ec0a65bfc4bf25d17ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } } peers { permanent_uuid: "28d3927fd978440dbea6a4d20ef45851" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } } peers { permanent_uuid: "2baa988c223b4c6aad408da9fd0c92b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34069 } }
W20260620 03:01:58.250253 13604 tablet.cc:2369] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:01:58.250834 13557 raft_consensus.cc:2454] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2baa988c223b4c6aad408da9fd0c92b0 in term 1.
I20260620 03:01:58.251530 13558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b572767e7c8344c5a4f49d9748dc17c1" candidate_uuid: "2baa988c223b4c6aad408da9fd0c92b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28d3927fd978440dbea6a4d20ef45851" is_pre_election: true
I20260620 03:01:58.251842 13558 raft_consensus.cc:2362] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 2baa988c223b4c6aad408da9fd0c92b0 in term 1. Re-sending same reply.
I20260620 03:01:58.256217 13260 catalog_manager.cc:5495] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2baa988c223b4c6aad408da9fd0c92b0 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "2baa988c223b4c6aad408da9fd0c92b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee2b5240a8b4ec0a65bfc4bf25d17ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28d3927fd978440dbea6a4d20ef45851" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2baa988c223b4c6aad408da9fd0c92b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34069 } health_report { overall_health: HEALTHY } } }
W20260620 03:01:58.271612 13460 tablet.cc:2369] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:01:58.284930 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:58.278305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.0.97.3:34069)
0620 03:01:58.278616 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:58.278623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:58.278893 (+ 270us) server_negotiation.cc:407] Connection header received
0620 03:01:58.278999 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:58.279003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:58.279085 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:58.279203 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:01:58.279693 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.280398 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.282412 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.282880 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.283023 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:58.283583 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:01:58.284105 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:01:58.284216 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:58.284475 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 03:01:58.284572 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260620 03:01:58.295753 13414 raft_consensus.cc:1261] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef [term 1 FOLLOWER]: Refusing update from remote peer 2baa988c223b4c6aad408da9fd0c92b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:01:58.296767 13557 raft_consensus.cc:1261] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851 [term 1 FOLLOWER]: Refusing update from remote peer 2baa988c223b4c6aad408da9fd0c92b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:01:58.297947 13768 consensus_queue.cc:1035] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28d3927fd978440dbea6a4d20ef45851" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:01:58.299564 13766 consensus_queue.cc:1035] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ee2b5240a8b4ec0a65bfc4bf25d17ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:01:58.316915 13778 mvcc.cc:205] Tried to move back new op lower bound from 7298762826927116288 to 7298762826747863040. Current Snapshot: MvccSnapshot[applied={T|T < 7298762826927116288}]
I20260620 03:01:58.345016 13261 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:01:58.345341 13261 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:01:58.349952 13261 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f464c6ca77e4a298dd81f9a78794dff: Sending DeleteTablet for 3 replicas of tablet b572767e7c8344c5a4f49d9748dc17c1
I20260620 03:01:58.351298 13538 tablet_service.cc:1514] Processing DeleteTablet for tablet b572767e7c8344c5a4f49d9748dc17c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:01:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53320
I20260620 03:01:58.351300 13394 tablet_service.cc:1514] Processing DeleteTablet for tablet b572767e7c8344c5a4f49d9748dc17c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:01:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56698
I20260620 03:01:58.352299 13788 tablet_replica.cc:330] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef: stopping tablet replica
I20260620 03:01:58.352901 13789 tablet_replica.cc:330] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851: stopping tablet replica
I20260620 03:01:58.353678 13672 tablet_service.cc:1514] Processing DeleteTablet for tablet b572767e7c8344c5a4f49d9748dc17c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:01:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53880
I20260620 03:01:58.361583 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 13298
I20260620 03:01:58.364473 13788 raft_consensus.cc:2229] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:01:58.364893 13789 raft_consensus.cc:2229] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:01:58.365590 13789 raft_consensus.cc:2258] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:01:58.366164 13788 raft_consensus.cc:2258] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:01:58.368790 13788 ts_tablet_manager.cc:1905] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:01:58.374123 13789 ts_tablet_manager.cc:1905] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:01:58.376513 13788 ts_tablet_manager.cc:1918] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:01:58.376760 13788 log.cc:1196] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/b572767e7c8344c5a4f49d9748dc17c1
I20260620 03:01:58.378726 13788 ts_tablet_manager.cc:1939] T b572767e7c8344c5a4f49d9748dc17c1 P 6ee2b5240a8b4ec0a65bfc4bf25d17ef: Deleting consensus metadata
I20260620 03:01:58.379712 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:58.362586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56702 (local address 127.0.97.1:39043)
0620 03:01:58.362802 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:58.362811 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:58.363351 (+ 540us) server_negotiation.cc:407] Connection header received
0620 03:01:58.363552 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:58.363560 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:58.363666 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:58.363779 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:58.365440 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.366625 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.368471 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.369094 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.369248 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:58.372516 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:58.372532 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:58.372552 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:58.372597 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:58.376553 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:58.377389 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:58.377399 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:58.377405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:58.377586 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:58.377992 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:58.377997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:58.378000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:58.378515 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:58.378859 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:58.379118 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 03:01:58.379324 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 03:01:58.380430 13454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:01:58.381181 13245 catalog_manager.cc:4841] TS 6ee2b5240a8b4ec0a65bfc4bf25d17ef (127.0.97.1:39043): tablet b572767e7c8344c5a4f49d9748dc17c1 (table test-table [id=d470c3a24159480b997058e81d8b6184]) successfully deleted
I20260620 03:01:58.391923 13789 ts_tablet_manager.cc:1918] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:01:58.392180 13789 log.cc:1196] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/b572767e7c8344c5a4f49d9748dc17c1
I20260620 03:01:58.394083 13790 tablet_replica.cc:330] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0: stopping tablet replica
I20260620 03:01:58.394788 13790 raft_consensus.cc:2229] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:01:58.395678 13790 raft_consensus.cc:2258] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:01:58.392870 13789 ts_tablet_manager.cc:1939] T b572767e7c8344c5a4f49d9748dc17c1 P 28d3927fd978440dbea6a4d20ef45851: Deleting consensus metadata
I20260620 03:01:58.398883 13790 ts_tablet_manager.cc:1905] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:01:58.408329 13246 catalog_manager.cc:4841] TS 28d3927fd978440dbea6a4d20ef45851 (127.0.97.2:37789): tablet b572767e7c8344c5a4f49d9748dc17c1 (table test-table [id=d470c3a24159480b997058e81d8b6184]) successfully deleted
I20260620 03:01:58.412127 13790 ts_tablet_manager.cc:1918] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:01:58.412318 13790 log.cc:1196] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/b572767e7c8344c5a4f49d9748dc17c1
I20260620 03:01:58.412914 13790 ts_tablet_manager.cc:1939] T b572767e7c8344c5a4f49d9748dc17c1 P 2baa988c223b4c6aad408da9fd0c92b0: Deleting consensus metadata
I20260620 03:01:58.414789 13248 catalog_manager.cc:4841] TS 2baa988c223b4c6aad408da9fd0c92b0 (127.0.97.3:34069): tablet b572767e7c8344c5a4f49d9748dc17c1 (table test-table [id=d470c3a24159480b997058e81d8b6184]) successfully deleted
I20260620 03:01:58.556814 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 13298
I20260620 03:01:58.580045 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 13463
I20260620 03:01:58.603225 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:58.580891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53334 (local address 127.0.97.2:37789)
0620 03:01:58.581064 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:58.581070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:58.581162 (+ 92us) server_negotiation.cc:407] Connection header received
0620 03:01:58.581353 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:58.581358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:58.581455 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:58.581563 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:58.583304 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.584429 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.586350 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.586979 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.587109 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:58.595796 (+ 8687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:58.595817 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:58.595839 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:58.595885 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:58.599905 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:58.600839 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:58.600849 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:58.600853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:58.601022 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:58.601474 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:58.601479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:58.601482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:58.601981 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:58.602263 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:58.602670 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 03:01:58.602883 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:01:58.603982 13598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:01:58.729174 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 13463
I20260620 03:01:58.746762 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 13607
I20260620 03:01:58.762913 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:01:58.747586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53894 (local address 127.0.97.3:34069)
0620 03:01:58.747745 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 03:01:58.747749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:01:58.747943 (+ 194us) server_negotiation.cc:407] Connection header received
0620 03:01:58.748131 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:01:58.748134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:01:58.748194 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:01:58.748280 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:01:58.750100 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.750876 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.752708 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:01:58.753166 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:01:58.753262 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:01:58.756514 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:01:58.756526 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:01:58.756543 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:01:58.756583 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:01:58.759975 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:58.760768 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:58.760774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:58.760777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:58.760913 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:01:58.761355 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:01:58.761359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:01:58.761361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:01:58.761831 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:01:58.762046 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 03:01:58.762420 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 03:01:58.762575 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 03:01:58.763594 13732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:01:58.887691 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 13607
I20260620 03:01:58.907984 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 13228
I20260620 03:01:58.909252 13291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:01:59.020282 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 13228
I20260620 03:01:59.045660 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 03:01:59.380939 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 03:01:59.381171 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 03:01:59.726626 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 03:01:59.726 UTC [12280] LOG: received smart shutdown request
2026-06-20 03:01:59.730 UTC [12280] LOG: background worker "logical replication launcher" (PID 12289) exited with exit code 1
2026-06-20 03:01:59.730 UTC [12284] LOG: shutting down
2026-06-20 03:01:59.903 UTC [12280] LOG: database system is shut down
2026-06-20T03:01:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52854 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:02:00 dist-test-slave-f1j0 krb5kdc[13803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:02:00 dist-test-slave-f1j0 krb5kdc[13803](info): set up 2 sockets
Jun 20 03:02:00 dist-test-slave-f1j0 krb5kdc[13803](info): commencing operation
krb5kdc: starting...
W20260620 03:02:01.150041 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:02:01 dist-test-slave-f1j0 krb5kdc[13803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:02:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:02:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:02:01.388854 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44051
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44051,127.0.97.61:43963,127.0.97.60:39107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:02:01.725663 13816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:01.726037 13816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:01.726214 13816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:01.733546 13816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:02:01.733738 13816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:01.733852 13816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:01.733960 13816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:02:01.734050 13816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:02:01.744035 13816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44051,127.0.97.61:43963,127.0.97.60:39107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:01.745503 13816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:01.747030 13816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:01.758793 13821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:01.758867 13822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:01.758893 13824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:01.759949 13816 server_base.cc:1034] running on GCE node
I20260620 03:02:01.760705 13816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:01.762135 13816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:01.763372 13816 hybrid_clock.cc:648] HybridClock initialized: now 1781924521763348 us; error 57 us; skew 500 ppm
I20260620 03:02:01.763720 13816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:01 dist-test-slave-f1j0 krb5kdc[13803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:01.771437 13816 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:02:01.773036 13816 webserver.cc:460] Webserver started at http://127.0.97.62:39131/ using document root <none> and password file <none>
I20260620 03:02:01.773813 13816 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:01.773923 13816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:01.774219 13816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:02:01.777238 13816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8"
format_stamp: "Formatted at 2026-06-20 03:02:01 on dist-test-slave-f1j0"
server_key: "cb046bd64125d2997d85202346967d7a"
server_key_iv: "a84c56d36f6732930ddd455d68609331"
server_key_version: "encryptionkey@0"
I20260620 03:02:01.778046 13816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8"
format_stamp: "Formatted at 2026-06-20 03:02:01 on dist-test-slave-f1j0"
server_key: "cb046bd64125d2997d85202346967d7a"
server_key_iv: "a84c56d36f6732930ddd455d68609331"
server_key_version: "encryptionkey@0"
I20260620 03:02:01.783099 13816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:02:01.786576 13831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:01.788002 13816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 03:02:01.788172 13816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8"
format_stamp: "Formatted at 2026-06-20 03:02:01 on dist-test-slave-f1j0"
server_key: "cb046bd64125d2997d85202346967d7a"
server_key_iv: "a84c56d36f6732930ddd455d68609331"
server_key_version: "encryptionkey@0"
I20260620 03:02:01.788353 13816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:01.826062 13816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:01.830184 13816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:01.830428 13816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:01.848886 13816 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44051
I20260620 03:02:01.848922 13882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44051 every 8 connection(s)
I20260620 03:02:01.850349 13816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:02:01.851706 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 13816
I20260620 03:02:01.851934 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:02:01.852568 388 external_mini_cluster.cc:1428] Setting key e12e41fc6b0ff8b357af0a096cbc5750
I20260620 03:02:01.854499 13883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:02:01.861815 13883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Jun 20 03:02:01 dist-test-slave-f1j0 krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:02:01.886993 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:01.866999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47094 (local address 127.0.97.62:44051)
0620 03:02:01.867740 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:01.867750 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:01.867810 (+ 60us) server_negotiation.cc:407] Connection header received
0620 03:02:01.869127 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:01.869137 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:01.869455 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:01.869710 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:01.870978 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:01.873072 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:01.874721 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:01.875479 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:01.875652 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:01.879490 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:01.879499 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:01.879513 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:01.879544 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:01.882453 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:01.883177 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:01.883192 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:01.883195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:01.883346 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:01.883737 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:01.883741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:01.883746 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:01.884283 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:01.884499 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:01.885033 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 03:02:01.885381 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":508,"thread_start_us":335,"threads_started":1}
I20260620 03:02:01.887010 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:01.865710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44051 (local address 127.0.0.1:47094)
0620 03:02:01.866546 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:01.866593 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:01.868652 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:01.869743 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:01.869748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:01.870407 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:01.870761 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:01.870779 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:01.873264 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:01.873268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:01.874553 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:01.874567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:01.875732 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:01.875737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:01.875880 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:01.876604 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:01.876627 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:01.879216 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:01.882604 (+ 3388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:01.882617 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:01.882631 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:01.883039 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:01.883455 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:01.883462 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:01.883465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:01.883622 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:01.884433 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:01.884441 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:01.884658 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 03:02:01.884911 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 03:02:01.885495 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":143,"thread_start_us":358,"threads_started":1}
I20260620 03:02:01.891036 13883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:02:01.892294 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:01.891814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43963 (local address 127.0.0.1:59358)
0620 03:02:01.892021 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:01.892144 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260620 03:02:01.892570 13832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43963: connect: Connection refused (error 111)
I20260620 03:02:01.894362 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:01.893970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43963 (local address 127.0.0.1:59368)
0620 03:02:01.894117 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:01.894210 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 03:02:01.894555 13883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43963: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43963: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 03:02:01.905706 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43963
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44051,127.0.97.61:43963,127.0.97.60:39107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:02:01.943881 13883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } attempt: 1
I20260620 03:02:01.945255 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:01.944588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43963 (local address 127.0.0.1:59384)
0620 03:02:01.944856 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:01.945056 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 03:02:01.946405 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:01.945972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43963 (local address 127.0.0.1:59394)
0620 03:02:01.946139 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:01.946242 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 03:02:01.946703 13883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43963: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43963: connect: Connection refused (error 111)
I20260620 03:02:02.015002 13883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } attempt: 2
I20260620 03:02:02.016286 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.015792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43963 (local address 127.0.0.1:59410)
0620 03:02:02.016049 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.016141 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 03:02:02.017508 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.017159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43963 (local address 127.0.0.1:59426)
0620 03:02:02.017296 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.017379 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 03:02:02.017802 13883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43963: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43963: connect: Connection refused (error 111)
I20260620 03:02:02.109062 13883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } attempt: 3
I20260620 03:02:02.110445 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.109983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43963 (local address 127.0.0.1:59438)
0620 03:02:02.110209 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.110303 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 03:02:02.111477 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.111107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43963 (local address 127.0.0.1:59452)
0620 03:02:02.111258 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.111344 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 03:02:02.111645 13883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43963: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43963: connect: Connection refused (error 111)
W20260620 03:02:02.232913 13890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:02.233261 13890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:02.233429 13890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:02.240741 13890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:02:02.240895 13890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:02.241012 13890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:02.241109 13890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:02:02.241204 13890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:02:02.251278 13890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44051,127.0.97.61:43963,127.0.97.60:39107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:02.252672 13890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:02.254072 13890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 03:02:02.254910 13883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } attempt: 4
I20260620 03:02:02.256181 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.255632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43963 (local address 127.0.0.1:59456)
0620 03:02:02.255960 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.256048 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260620 03:02:02.257237 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.256902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43963 (local address 127.0.0.1:59458)
0620 03:02:02.257035 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.257124 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 03:02:02.257505 13883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43963: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43963: connect: Connection refused (error 111)
W20260620 03:02:02.266562 13896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:02.266628 13898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:02.267201 13895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:02.267999 13890 server_base.cc:1034] running on GCE node
I20260620 03:02:02.268818 13890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:02.270310 13890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:02.271513 13890 hybrid_clock.cc:648] HybridClock initialized: now 1781924522271470 us; error 32 us; skew 500 ppm
I20260620 03:02:02.271901 13890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:02 dist-test-slave-f1j0 krb5kdc[13803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:02.275115 13890 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 03:02:02.276597 13890 webserver.cc:460] Webserver started at http://127.0.97.61:45013/ using document root <none> and password file <none>
I20260620 03:02:02.277293 13890 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:02.277434 13890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:02.277799 13890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:02:02.280643 13890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-1/data/instance:
uuid: "c5f5a36346f24e36813f3214a7a932d9"
format_stamp: "Formatted at 2026-06-20 03:02:02 on dist-test-slave-f1j0"
server_key: "a1285b47b2e9346a0dde891e02f2a7a1"
server_key_iv: "8e06d2f224b14aa029f4a72e2623eda6"
server_key_version: "encryptionkey@0"
I20260620 03:02:02.281400 13890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-1/wal/instance:
uuid: "c5f5a36346f24e36813f3214a7a932d9"
format_stamp: "Formatted at 2026-06-20 03:02:02 on dist-test-slave-f1j0"
server_key: "a1285b47b2e9346a0dde891e02f2a7a1"
server_key_iv: "8e06d2f224b14aa029f4a72e2623eda6"
server_key_version: "encryptionkey@0"
I20260620 03:02:02.286741 13890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:02:02.290231 13905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:02.291591 13890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:02:02.291790 13890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-1/wal
uuid: "c5f5a36346f24e36813f3214a7a932d9"
format_stamp: "Formatted at 2026-06-20 03:02:02 on dist-test-slave-f1j0"
server_key: "a1285b47b2e9346a0dde891e02f2a7a1"
server_key_iv: "8e06d2f224b14aa029f4a72e2623eda6"
server_key_version: "encryptionkey@0"
I20260620 03:02:02.292001 13890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:02.308619 13890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:02.312572 13890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:02.312868 13890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:02.331624 13890 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:43963
I20260620 03:02:02.331663 13956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:43963 every 8 connection(s)
I20260620 03:02:02.333011 13890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-1/data/info.pb
I20260620 03:02:02.336722 13957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:02:02.337990 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 13890
I20260620 03:02:02.338197 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-1/wal/instance
I20260620 03:02:02.338770 388 external_mini_cluster.cc:1428] Setting key 8b02716d98c31e4027f4a33428d88d8b
I20260620 03:02:02.346836 13957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 20 03:02:02 dist-test-slave-f1j0 krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:02:02.370855 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.349759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47106 (local address 127.0.97.62:44051)
0620 03:02:02.349948 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:02.349953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:02.351170 (+ 1217us) server_negotiation.cc:407] Connection header received
0620 03:02:02.353121 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:02.353127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:02.353223 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:02.353377 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:02.355043 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.355796 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.357468 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.357975 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.358112 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.364204 (+ 6092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:02.364209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:02.364213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:02.364249 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:02.367173 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.368220 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.368224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.368227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.368351 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.368841 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.368844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.368846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.369135 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:02.369330 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:02.370154 (+ 824us) server_negotiation.cc:299] Negotiation successful
0620 03:02:02.370426 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 03:02:02.371541 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.350219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44051 (local address 127.0.0.1:47106)
0620 03:02:02.350995 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.351032 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:02.352819 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:02.353642 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:02.353655 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:02.354277 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:02.354864 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:02.354881 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.355967 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.355971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:02.357308 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:02.357322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.358222 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.358230 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:02.358443 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.359144 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:02.359167 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:02.363914 (+ 4747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:02.367343 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:02.367356 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:02.367377 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:02.368059 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:02.368464 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:02.368471 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:02.368477 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:02.368723 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:02.369314 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:02.369325 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:02.369756 (+ 431us) client_negotiation.cc:769] Sending connection context
0620 03:02:02.369992 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 03:02:02.370388 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":323,"threads_started":1}
I20260620 03:02:02.373441 13957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:02:02 dist-test-slave-f1j0 krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:02:02.389786 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.374125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43963 (local address 127.0.0.1:59474)
0620 03:02:02.374279 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.374306 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:02.374405 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:02.375525 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:02.375533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:02.376062 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:02.376293 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:02.376307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.377656 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.377664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:02.379295 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:02.379312 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.380254 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.380266 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:02.380523 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.381130 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:02.381170 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:02.383643 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:02.387030 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:02.387037 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:02.387041 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:02.387425 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:02.387789 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:02.387796 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:02.387799 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:02.387991 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:02.388683 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:02.388687 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:02.388841 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 03:02:02.389007 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 03:02:02.389360 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 03:02:02.389884 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.374445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59474 (local address 127.0.97.61:43963)
0620 03:02:02.375091 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:02.375096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:02.375119 (+ 23us) server_negotiation.cc:407] Connection header received
0620 03:02:02.375160 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:02.375166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:02.375350 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:02.375465 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:02.376450 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.377460 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.379442 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.380087 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.380183 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.383853 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:02.383864 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:02.383877 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:02.383904 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:02.386898 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.387556 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.387565 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.387570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.387681 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.388105 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.388113 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.388118 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.388566 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:02.388782 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:02.389160 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 03:02:02.389487 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":499,"thread_start_us":368,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 03:02:02.392657 13957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:02:02.394024 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.393501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33582)
0620 03:02:02.393746 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.393878 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260620 03:02:02.394034 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39107
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44051,127.0.97.61:43963,127.0.97.60:39107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:02:02.394307 13908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
I20260620 03:02:02.395949 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.395607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33590)
0620 03:02:02.395758 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.395847 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 03:02:02.396153 13957 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39107: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
I20260620 03:02:02.445510 13957 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } attempt: 1
I20260620 03:02:02.446950 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.446349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33604)
0620 03:02:02.446621 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.446757 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 03:02:02.448288 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.447810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33612)
0620 03:02:02.448053 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.448146 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260620 03:02:02.448585 13957 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39107: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
I20260620 03:02:02.516865 13957 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } attempt: 2
I20260620 03:02:02.518266 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.517728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33624)
0620 03:02:02.517998 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.518108 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260620 03:02:02.519452 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.519059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33640)
0620 03:02:02.519250 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.519335 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260620 03:02:02.519791 13957 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39107: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
I20260620 03:02:02.556825 13883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } attempt: 5
Jun 20 03:02:02 dist-test-slave-f1j0 krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:02:02.571466 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.557791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59482 (local address 127.0.97.61:43963)
0620 03:02:02.557981 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:02.557985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:02.558005 (+ 20us) server_negotiation.cc:407] Connection header received
0620 03:02:02.558209 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:02.558294 (+ 85us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:02.558394 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:02.558493 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:02.559579 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.560268 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.561941 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.562436 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.562544 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.566208 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:02.566213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:02.566216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:02.566248 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:02.568854 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.569538 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.569542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.569544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.569714 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.570093 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.570098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.570101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.570387 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:02.570539 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:02.570916 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 03:02:02.571146 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 03:02:02.571466 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.557678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43963 (local address 127.0.0.1:59482)
0620 03:02:02.557926 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.557949 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:02.558080 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:02.558568 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:02.558573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:02.559156 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:02.559415 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:02.559430 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.560419 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.560424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:02.561773 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:02.561789 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.562597 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.562603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:02.562804 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.563494 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:02.563517 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:02.566002 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:02.568999 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:02.569005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:02.569010 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:02.569398 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:02.569834 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:02.569838 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:02.569842 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:02.569977 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:02.570504 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:02.570509 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:02.570670 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 03:02:02.570803 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 03:02:02.571087 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 03:02:02.572678 13883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:02:02.573786 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.573375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33652)
0620 03:02:02.573543 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.573657 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 03:02:02.574918 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.574540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33666)
0620 03:02:02.574713 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.574795 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 03:02:02.575178 13883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39107: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
I20260620 03:02:02.611021 13957 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } attempt: 3
I20260620 03:02:02.612182 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.611715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33670)
0620 03:02:02.611931 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.612026 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 03:02:02.613309 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.612836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33672)
0620 03:02:02.613089 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.613182 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260620 03:02:02.613615 13957 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39107: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
I20260620 03:02:02.626408 13883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } attempt: 1
I20260620 03:02:02.627573 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.627061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33678)
0620 03:02:02.627278 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.627383 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 03:02:02.628659 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.628252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33694)
0620 03:02:02.628402 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.628499 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 03:02:02.628892 13883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39107: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
I20260620 03:02:02.697134 13883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } attempt: 2
I20260620 03:02:02.698489 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.697870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33700)
0620 03:02:02.698123 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.698263 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 03:02:02.699710 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.699291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33702)
0620 03:02:02.699475 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.699564 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 03:02:02.699952 13883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39107: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
W20260620 03:02:02.733505 13964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:02.733875 13964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:02.734061 13964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:02.741287 13964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:02:02.741392 13964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:02.741482 13964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:02.741614 13964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:02:02.741679 13964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:02:02.751636 13964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44051,127.0.97.61:43963,127.0.97.60:39107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:02.752981 13964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:02.754439 13964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 03:02:02.756872 13957 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } attempt: 4
I20260620 03:02:02.758073 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.757495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33704)
0620 03:02:02.757774 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.757883 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 03:02:02.759150 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.758763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33710)
0620 03:02:02.758934 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.759025 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 03:02:02.759392 13957 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39107: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
W20260620 03:02:02.766801 13969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:02.766853 13970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:02.766871 13972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:02.768052 13964 server_base.cc:1034] running on GCE node
I20260620 03:02:02.768934 13964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:02.770550 13964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:02.771771 13964 hybrid_clock.cc:648] HybridClock initialized: now 1781924522771725 us; error 37 us; skew 500 ppm
I20260620 03:02:02.772151 13964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:02 dist-test-slave-f1j0 krb5kdc[13803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:02.775626 13964 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 03:02:02.777223 13964 webserver.cc:460] Webserver started at http://127.0.97.60:40449/ using document root <none> and password file <none>
I20260620 03:02:02.778010 13964 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:02.778147 13964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:02.778484 13964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:02:02.781468 13964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-2/data/instance:
uuid: "fd364c12e4b34733832f08e0202fd80c"
format_stamp: "Formatted at 2026-06-20 03:02:02 on dist-test-slave-f1j0"
server_key: "2231117293ee81fe0f316b32ee35dc84"
server_key_iv: "a1306381cd582376d4339ebc89e5dd26"
server_key_version: "encryptionkey@0"
I20260620 03:02:02.782367 13964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-2/wal/instance:
uuid: "fd364c12e4b34733832f08e0202fd80c"
format_stamp: "Formatted at 2026-06-20 03:02:02 on dist-test-slave-f1j0"
server_key: "2231117293ee81fe0f316b32ee35dc84"
server_key_iv: "a1306381cd582376d4339ebc89e5dd26"
server_key_version: "encryptionkey@0"
I20260620 03:02:02.787626 13964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:02:02.791316 13979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:02.792884 13964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 03:02:02.793035 13964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-2/wal
uuid: "fd364c12e4b34733832f08e0202fd80c"
format_stamp: "Formatted at 2026-06-20 03:02:02 on dist-test-slave-f1j0"
server_key: "2231117293ee81fe0f316b32ee35dc84"
server_key_iv: "a1306381cd582376d4339ebc89e5dd26"
server_key_version: "encryptionkey@0"
I20260620 03:02:02.793210 13964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:02.806207 13883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } attempt: 3
I20260620 03:02:02.807392 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.806801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33726)
0620 03:02:02.807056 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.807175 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 03:02:02.808492 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.808057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33740)
0620 03:02:02.808258 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.808358 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260620 03:02:02.808720 13883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39107: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39107: connect: Connection refused (error 111)
I20260620 03:02:02.809805 13964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:02.814347 13964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:02.814625 13964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:02.833698 13964 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:39107
I20260620 03:02:02.833746 14030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:39107 every 8 connection(s)
I20260620 03:02:02.835089 13964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-2/data/info.pb
I20260620 03:02:02.836575 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 13964
I20260620 03:02:02.836799 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781924320356866-388-0/minicluster-data/master-2/wal/instance
I20260620 03:02:02.837322 388 external_mini_cluster.cc:1428] Setting key 081b3b58b9c4abd4251b4118c41ff6ae
I20260620 03:02:02.839751 14031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:02:02.847900 14031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:02:02 dist-test-slave-f1j0 krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 03:02:02 dist-test-slave-f1j0 krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:02:02.862133 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.839271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33750 (local address 127.0.97.60:39107)
0620 03:02:02.840085 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:02.840097 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:02.840138 (+ 41us) server_negotiation.cc:407] Connection header received
0620 03:02:02.841768 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:02.841790 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:02.842183 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:02.842500 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:02.844254 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.845673 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.847499 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.848275 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.848462 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.853904 (+ 5442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:02.853913 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:02.853926 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:02.853968 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:02.857672 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.858535 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.858541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.858547 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.858758 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.859179 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.859182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.859184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.859708 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:02.860015 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:02.860410 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 03:02:02.860866 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"spinlock_wait_cycles":12672,"thread_start_us":387,"threads_started":1}
I20260620 03:02:02.867019 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.851051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47122 (local address 127.0.97.62:44051)
0620 03:02:02.851220 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:02.851224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:02.851949 (+ 725us) server_negotiation.cc:407] Connection header received
0620 03:02:02.852248 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:02.852253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:02.852314 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:02.852419 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:02.853517 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.854668 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.856810 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.857553 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.857708 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.861298 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:02.861302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:02.861305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:02.861336 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:02.864009 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.864726 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.864733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.864736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.864964 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.865463 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.865473 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.865477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.865862 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:02.866014 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:02.866609 (+ 595us) server_negotiation.cc:299] Negotiation successful
0620 03:02:02.866764 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 03:02:02.867305 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.851014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44051 (local address 127.0.0.1:47122)
0620 03:02:02.851801 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.851857 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:02.852089 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:02.852476 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:02.852482 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:02.853107 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:02.853374 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:02.853383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.854839 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.854843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:02.856662 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:02.856676 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.857757 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.857765 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:02.857947 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.858543 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:02.858565 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:02.861111 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:02.864130 (+ 3019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:02.864138 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:02.864143 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:02.864601 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:02.865107 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:02.865112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:02.865116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:02.865314 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:02.866032 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:02.866040 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:02.866297 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 03:02:02.866521 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 03:02:02.866846 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":376,"threads_started":1}
I20260620 03:02:02.868673 14031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:02:02 dist-test-slave-f1j0 krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 03:02:02 dist-test-slave-f1j0 krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:02:02.880756 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.865051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47134 (local address 127.0.97.62:44051)
0620 03:02:02.865743 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:02.865749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:02.865770 (+ 21us) server_negotiation.cc:407] Connection header received
0620 03:02:02.865813 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:02.865818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:02.865880 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:02.865977 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:02.867595 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.868442 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.869895 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.870304 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.870435 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.874724 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:02.874729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:02.874732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:02.874762 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:02.878076 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.878739 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.878745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.878748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.878871 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.879242 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.879248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.879251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.879532 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:02.879836 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:02.880102 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 03:02:02.880328 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":319,"threads_started":1}
I20260620 03:02:02.881558 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.869430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.0.97.61:43963)
0620 03:02:02.869627 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:02.869634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:02.869652 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:02:02.869704 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:02.869709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:02.869774 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:02.869888 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:02.870542 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.871211 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.872404 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.872817 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.872964 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.876327 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:02.876334 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:02.876338 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:02.876406 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:02.879017 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.879673 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.879679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.879682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.879838 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.880229 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.880236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.880239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.880524 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:02.880686 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:02.881137 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 03:02:02.881308 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 03:02:02.881698 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.869291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43963 (local address 127.0.0.1:59492)
0620 03:02:02.869433 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.869451 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:02.869542 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:02.869905 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:02.869910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:02.870259 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:02.870398 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:02.870406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.871364 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.871370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:02.872222 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:02.872232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.872977 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.872983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:02.873191 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.873773 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:02.873795 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:02.876108 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:02.879154 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:02.879160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:02.879164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:02.879544 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:02.879965 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:02.879970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:02.879973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:02.880122 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:02.880659 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:02.880666 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:02.880835 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 03:02:02.881040 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 03:02:02.881325 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 03:02:02.882781 14031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:02:02 dist-test-slave-f1j0 krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 03:02:02 dist-test-slave-f1j0 krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:02:02.897940 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.883736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33760)
0620 03:02:02.883999 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.884017 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:02.884127 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:02.884430 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:02.884435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:02.884865 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:02.884996 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:02.885003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.886212 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.886218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:02.887587 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:02.887600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.888528 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.888534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:02.888744 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.889183 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:02.889209 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:02.891768 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:02.895187 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:02.895192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:02.895196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:02.895709 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:02.896155 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:02.896160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:02.896164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:02.896296 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:02.896806 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:02.896831 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:02.896978 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 03:02:02.897167 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 03:02:02.897469 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260620 03:02:02.897996 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.883723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.0.97.60:39107)
0620 03:02:02.883881 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:02.883885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:02.884086 (+ 201us) server_negotiation.cc:407] Connection header received
0620 03:02:02.884235 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:02.884238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:02.884286 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:02.884370 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:02.885128 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.886045 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.887711 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.888354 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.888461 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.891965 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:02.891970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:02.891974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:02.892001 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:02.895055 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.895876 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.895880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.895884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.896038 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.896417 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.896421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.896425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.896694 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:02.897045 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:02.897258 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 03:02:02.897662 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":51}
I20260620 03:02:02.899559 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.882566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.0.97.61:43963)
0620 03:02:02.882718 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:02.882723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:02.882846 (+ 123us) server_negotiation.cc:407] Connection header received
0620 03:02:02.882932 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:02.882939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:02.883007 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:02.883105 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:02.884709 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.885725 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.887521 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.888125 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.888249 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.893277 (+ 5028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:02.893282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:02.893286 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:02.893315 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:02.896252 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.896959 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.896967 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.896971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.897106 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.897485 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.897491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.897495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.897835 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:02.898108 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:02.899052 (+ 944us) server_negotiation.cc:299] Negotiation successful
0620 03:02:02.899288 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 03:02:02.903800 14031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c: Bootstrap starting.
I20260620 03:02:02.904361 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:02:02.907104 14031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c: Neither blocks nor log segments found. Creating new log.
I20260620 03:02:02.908147 14031 log.cc:824] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:02.915165 14031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c: No bootstrap required, opened a new log
I20260620 03:02:02.926676 14031 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } } peers { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } } peers { permanent_uuid: "fd364c12e4b34733832f08e0202fd80c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } }
I20260620 03:02:02.927265 14031 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:02:02.927459 14031 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd364c12e4b34733832f08e0202fd80c, State: Initialized, Role: FOLLOWER
I20260620 03:02:02.928356 14031 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } } peers { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } } peers { permanent_uuid: "fd364c12e4b34733832f08e0202fd80c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } }
I20260620 03:02:02.929507 14045 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } } peers { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } } peers { permanent_uuid: "fd364c12e4b34733832f08e0202fd80c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } } }
I20260620 03:02:02.930078 14045 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:02:02.930924 14031 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:02:02.935757 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.909289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33764 (local address 127.0.97.60:39107)
0620 03:02:02.909454 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:02.909467 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:02.916782 (+ 7315us) server_negotiation.cc:407] Connection header received
0620 03:02:02.916840 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:02.916844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:02.916917 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:02.917007 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:02.919350 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.920464 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.922805 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.923511 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.923601 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.927455 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:02.927462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:02.927468 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:02.927509 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:02.931190 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.932154 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.932159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.932163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.932373 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.932939 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.932946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.932952 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.933396 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:02.933712 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:02.934940 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0620 03:02:02.935222 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 03:02:02.936432 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.909717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47140 (local address 127.0.97.62:44051)
0620 03:02:02.909891 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:02.909896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:02.915962 (+ 6066us) server_negotiation.cc:407] Connection header received
0620 03:02:02.916140 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:02.916149 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:02.916251 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:02.916358 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:02.919109 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.920580 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.923511 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.924394 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.924515 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.929117 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:02.929126 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:02.929130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:02.929173 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:02.933078 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.933867 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.933872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.933876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.934019 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.934535 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.934541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.934544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.934886 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:02.935116 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:02.935851 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 03:02:02.936079 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 03:02:02.946954 14056 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 03:02:02.947157 14056 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 03:02:02.947839 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.916743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59502 (local address 127.0.97.61:43963)
0620 03:02:02.916925 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:02.916931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:02.930474 (+ 13543us) server_negotiation.cc:407] Connection header received
0620 03:02:02.930742 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:02.930748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:02.930844 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:02.930916 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:02.932812 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.933965 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.936045 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.936711 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.936814 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.940208 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:02.940214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:02.940217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:02.940253 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:02.943188 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.943968 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.943973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.943977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.944144 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:02.946037 (+ 1893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:02.946043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:02.946047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:02.946422 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:02.946643 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:02.947242 (+ 599us) server_negotiation.cc:299] Negotiation successful
0620 03:02:02.947480 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:02:02.943986 13964 master_runner.cc:186] Error getting master registration for 127.0.97.62:44051: OK, instance_id { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" instance_seqno: 1781924521843545 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 03:02:02.953301 13964 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 03:02:02.985935 13883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } attempt: 4
Jun 20 03:02:02 dist-test-slave-f1j0 krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 03:02:03.002336 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.986672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33768 (local address 127.0.97.60:39107)
0620 03:02:02.986856 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:02.986862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:02.986884 (+ 22us) server_negotiation.cc:407] Connection header received
0620 03:02:02.987026 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:02.987032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:02.987131 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:02.987243 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:02.988230 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.989434 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.991220 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.991648 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.991746 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.995599 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:02.995604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:02.995607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:02.995658 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:02.999524 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:03.000312 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:03.000316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:03.000318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:03.000490 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:03.000900 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:03.000903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:03.000905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:03.001156 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:03.001268 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:03.001788 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 03:02:03.002022 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 03:02:03.002331 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:02.986590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33768)
0620 03:02:02.986777 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:02.986804 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:02.986903 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:02.987274 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:02.987277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:02.987817 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:02.988053 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:02.988066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.989604 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.989610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:02.991089 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:02.991104 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:02.991796 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:02.991802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:02.992022 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:02.992671 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:02.992701 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:02.995406 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:02.999665 (+ 4259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:02.999671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:02.999677 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:03.000199 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:03.000599 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:03.000605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:03.000608 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:03.000791 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:03.001279 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:03.001285 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:03.001455 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 03:02:03.001643 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 03:02:03.001959 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 03:02:03.007798 13883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8: Bootstrap starting.
I20260620 03:02:03.010943 13883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8: Neither blocks nor log segments found. Creating new log.
I20260620 03:02:03.011955 13883 log.cc:824] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:03.014709 13883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8: No bootstrap required, opened a new log
I20260620 03:02:03.022583 13883 raft_consensus.cc:348] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } } peers { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } } peers { permanent_uuid: "fd364c12e4b34733832f08e0202fd80c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } }
I20260620 03:02:03.022933 13883 raft_consensus.cc:374] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:02:03.023068 13883 raft_consensus.cc:729] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f05d4979b4fe41fe8a0afc0323a9a7e8, State: Initialized, Role: FOLLOWER
I20260620 03:02:03.023648 13883 consensus_queue.cc:260] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } } peers { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } } peers { permanent_uuid: "fd364c12e4b34733832f08e0202fd80c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } }
I20260620 03:02:03.024622 14059 sys_catalog.cc:455] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } } peers { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } } peers { permanent_uuid: "fd364c12e4b34733832f08e0202fd80c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } } }
I20260620 03:02:03.025022 14059 sys_catalog.cc:458] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:02:03.025938 13883 sys_catalog.cc:564] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 03:02:03.037050 14070 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 03:02:03.037278 14070 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 03:02:03.035467 13816 master_runner.cc:186] Error getting master registration for 127.0.97.61:43963: OK, instance_id { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" instance_seqno: 1781924522326248 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 03:02:03.046903 13816 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 03:02:03.058625 13957 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } attempt: 5
W20260620 03:02:03.066883 13985 tablet.cc:2369] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 03:02:03 dist-test-slave-f1j0 krb5kdc[13803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 03:02:03.078088 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:03.059407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.0.97.60:39107)
0620 03:02:03.059589 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:03.059595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:03.059664 (+ 69us) server_negotiation.cc:407] Connection header received
0620 03:02:03.059858 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:03.059864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:03.059949 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:03.060069 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:03.061037 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:03.064050 (+ 3013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:03.065856 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:03.066548 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:03.066680 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:03.071015 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:03.071022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:03.071028 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:03.071070 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:03.074904 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:03.075740 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:03.075746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:03.075750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:03.075919 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:03.076381 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:03.076387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:03.076390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:03.076764 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:03.076909 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:03.077439 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 03:02:03.077698 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 03:02:03.078109 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:03.059281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39107 (local address 127.0.0.1:33776)
0620 03:02:03.059567 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:03.059588 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:03.059721 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:03.060110 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:03.060115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:03.060640 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:03.060863 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:03.060876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:03.064197 (+ 3321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:03.064205 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:03.065698 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:03.065714 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:03.066930 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:03.066937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:03.067223 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:03.067907 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:03.067943 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:03.070826 (+ 2883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:03.075048 (+ 4222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:03.075054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:03.075059 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:03.075602 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:03.076048 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:03.076053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:03.076057 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:03.076226 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:03.076974 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:03.076979 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:03.077142 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 03:02:03.077309 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 03:02:03.077689 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
W20260620 03:02:03.083256 13837 tablet.cc:2369] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:02:03.084750 13957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9: Bootstrap starting.
I20260620 03:02:03.089710 13957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9: Neither blocks nor log segments found. Creating new log.
I20260620 03:02:03.091046 13957 log.cc:824] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:03.094657 13957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9: No bootstrap required, opened a new log
I20260620 03:02:03.106091 13957 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } } peers { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } } peers { permanent_uuid: "fd364c12e4b34733832f08e0202fd80c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } }
I20260620 03:02:03.106505 13957 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:02:03.106630 13957 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5f5a36346f24e36813f3214a7a932d9, State: Initialized, Role: FOLLOWER
I20260620 03:02:03.107316 13957 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } } peers { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } } peers { permanent_uuid: "fd364c12e4b34733832f08e0202fd80c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } }
I20260620 03:02:03.108296 14073 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } } peers { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } } peers { permanent_uuid: "fd364c12e4b34733832f08e0202fd80c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } } }
I20260620 03:02:03.108788 14073 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:02:03.109755 13957 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:02:03.129015 13890 master_runner.cc:263] No leader master found from master c5f5a36346f24e36813f3214a7a932d9
I20260620 03:02:03.129204 13890 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 03:02:03.132372 14084 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 03:02:03.132514 14084 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 03:02:03.218490 14045 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:02:03.218859 14045 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } } peers { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } } peers { permanent_uuid: "fd364c12e4b34733832f08e0202fd80c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } }
I20260620 03:02:03.220582 14045 leader_election.cc:288] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f05d4979b4fe41fe8a0afc0323a9a7e8 (127.0.97.62:44051), c5f5a36346f24e36813f3214a7a932d9 (127.0.97.61:43963)
I20260620 03:02:03.221712 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd364c12e4b34733832f08e0202fd80c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5f5a36346f24e36813f3214a7a932d9" is_pre_election: true
I20260620 03:02:03.222311 13932 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd364c12e4b34733832f08e0202fd80c in term 0.
I20260620 03:02:03.222952 13980 leader_election.cc:302] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5f5a36346f24e36813f3214a7a932d9, fd364c12e4b34733832f08e0202fd80c; no voters:
I20260620 03:02:03.223955 13858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd364c12e4b34733832f08e0202fd80c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" is_pre_election: true
I20260620 03:02:03.224499 13858 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd364c12e4b34733832f08e0202fd80c in term 0.
I20260620 03:02:03.225335 14045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:02:03.225517 14045 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:02:03.225653 14045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:02:03.229470 14045 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } } peers { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } } peers { permanent_uuid: "fd364c12e4b34733832f08e0202fd80c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } }
I20260620 03:02:03.230587 13858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd364c12e4b34733832f08e0202fd80c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8"
I20260620 03:02:03.230808 13858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:02:03.234236 13858 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd364c12e4b34733832f08e0202fd80c in term 1.
I20260620 03:02:03.234680 13980 leader_election.cc:302] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f05d4979b4fe41fe8a0afc0323a9a7e8, fd364c12e4b34733832f08e0202fd80c; no voters:
I20260620 03:02:03.236811 14045 leader_election.cc:288] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [CANDIDATE]: Term 1 election: Requested vote from peers f05d4979b4fe41fe8a0afc0323a9a7e8 (127.0.97.62:44051), c5f5a36346f24e36813f3214a7a932d9 (127.0.97.61:43963)
I20260620 03:02:03.237389 14045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:02:03.237782 14045 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [term 1 LEADER]: Becoming Leader. State: Replica: fd364c12e4b34733832f08e0202fd80c, State: Running, Role: LEADER
I20260620 03:02:03.238613 14045 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } } peers { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } } peers { permanent_uuid: "fd364c12e4b34733832f08e0202fd80c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } }
I20260620 03:02:03.251302 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd364c12e4b34733832f08e0202fd80c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5f5a36346f24e36813f3214a7a932d9"
I20260620 03:02:03.251579 13932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:02:03.255223 13932 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd364c12e4b34733832f08e0202fd80c in term 1.
I20260620 03:02:03.258540 14086 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd364c12e4b34733832f08e0202fd80c. Latest consensus state: current_term: 1 leader_uuid: "fd364c12e4b34733832f08e0202fd80c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } } peers { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } } peers { permanent_uuid: "fd364c12e4b34733832f08e0202fd80c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } } }
I20260620 03:02:03.259007 14086 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [sys.catalog]: This master's current role is: LEADER
I20260620 03:02:03.265944 14088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:02:03.268253 14088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:02:03.285717 13932 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9 [term 1 FOLLOWER]: Refusing update from remote peer fd364c12e4b34733832f08e0202fd80c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:02:03.289245 14086 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:02:03.296854 13858 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8 [term 1 FOLLOWER]: Refusing update from remote peer fd364c12e4b34733832f08e0202fd80c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:02:03.297935 14086 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 03:02:03.315569 13911 tablet.cc:2369] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:02:03.338265 14073 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd364c12e4b34733832f08e0202fd80c. Latest consensus state: current_term: 1 leader_uuid: "fd364c12e4b34733832f08e0202fd80c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } } peers { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } } peers { permanent_uuid: "fd364c12e4b34733832f08e0202fd80c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } } }
I20260620 03:02:03.338593 14073 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:02:03.342855 14089 mvcc.cc:205] Tried to move back new op lower bound from 7298762847358554112 to 7298762847198216192. Current Snapshot: MvccSnapshot[applied={T|T < 7298762847358554112}]
I20260620 03:02:03.345120 14045 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd364c12e4b34733832f08e0202fd80c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } } peers { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } } peers { permanent_uuid: "fd364c12e4b34733832f08e0202fd80c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } } }
I20260620 03:02:03.345458 14045 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [sys.catalog]: This master's current role is: LEADER
I20260620 03:02:03.350775 14088 catalog_manager.cc:1347] Generated new cluster ID: 0301cd42486d49199c10dd8831c435af
I20260620 03:02:03.350914 14088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:02:03.356554 14073 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd364c12e4b34733832f08e0202fd80c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } } peers { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } } peers { permanent_uuid: "fd364c12e4b34733832f08e0202fd80c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } } }
I20260620 03:02:03.356819 14073 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:02:03.362694 14059 sys_catalog.cc:455] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd364c12e4b34733832f08e0202fd80c. Latest consensus state: current_term: 1 leader_uuid: "fd364c12e4b34733832f08e0202fd80c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } } peers { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } } peers { permanent_uuid: "fd364c12e4b34733832f08e0202fd80c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } } }
I20260620 03:02:03.363016 14059 sys_catalog.cc:458] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:02:03.365815 14086 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd364c12e4b34733832f08e0202fd80c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } } peers { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } } peers { permanent_uuid: "fd364c12e4b34733832f08e0202fd80c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } } }
I20260620 03:02:03.366087 14086 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c [sys.catalog]: This master's current role is: LEADER
I20260620 03:02:03.368332 14091 mvcc.cc:205] Tried to move back new op lower bound from 7298762847358554112 to 7298762847198216192. Current Snapshot: MvccSnapshot[applied={T|T < 7298762847358554112}]
I20260620 03:02:03.378149 14059 sys_catalog.cc:455] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd364c12e4b34733832f08e0202fd80c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05d4979b4fe41fe8a0afc0323a9a7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44051 } } peers { permanent_uuid: "c5f5a36346f24e36813f3214a7a932d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43963 } } peers { permanent_uuid: "fd364c12e4b34733832f08e0202fd80c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39107 } } }
I20260620 03:02:03.378458 14059 sys_catalog.cc:458] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:02:03.417361 14088 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:02:03.419514 14088 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:02:03.442986 14088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd364c12e4b34733832f08e0202fd80c: Generated new TSK 0
I20260620 03:02:03.444268 14088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:02:04.039309 14070 catalog_manager.cc:1259] Loaded cluster ID: 0301cd42486d49199c10dd8831c435af
I20260620 03:02:04.039408 14070 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8: loading cluster ID for follower catalog manager: success
I20260620 03:02:04.041750 14070 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8: acquiring CA information for follower catalog manager: success
I20260620 03:02:04.043418 14070 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f05d4979b4fe41fe8a0afc0323a9a7e8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 03:02:04.134531 14084 catalog_manager.cc:1259] Loaded cluster ID: 0301cd42486d49199c10dd8831c435af
I20260620 03:02:04.134637 14084 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9: loading cluster ID for follower catalog manager: success
I20260620 03:02:04.136981 14084 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9: acquiring CA information for follower catalog manager: success
I20260620 03:02:04.138756 14084 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c5f5a36346f24e36813f3214a7a932d9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 03:02:04.536219 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 13816
I20260620 03:02:04.537249 13878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:04.649303 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 13816
I20260620 03:02:04.669829 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 13890
I20260620 03:02:04.670907 13952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:04.778870 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 13890
I20260620 03:02:04.799293 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 13964
I20260620 03:02:04.800392 14026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:04.909054 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 13964
2026-06-20T03:02:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4876 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:02:05 dist-test-slave-f1j0 krb5kdc[14106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:02:05 dist-test-slave-f1j0 krb5kdc[14106](info): set up 2 sockets
Jun 20 03:02:05 dist-test-slave-f1j0 krb5kdc[14106](info): commencing operation
krb5kdc: starting...
W20260620 03:02:06.010027 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:02:06 dist-test-slave-f1j0 krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:02:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:02:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:02:06.245311 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36753
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:02:06.577383 14119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:06.577760 14119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:06.577868 14119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:06.585027 14119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:02:06.585135 14119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:06.585182 14119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:06.585239 14119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:02:06.585282 14119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:02:06.595193 14119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:06.596554 14119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:06.597949 14119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:06.610968 14125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:06.611034 14124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:06.611061 14127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:06.611665 14119 server_base.cc:1034] running on GCE node
I20260620 03:02:06.612493 14119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:06.614027 14119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:06.615267 14119 hybrid_clock.cc:648] HybridClock initialized: now 1781924526615202 us; error 57 us; skew 500 ppm
I20260620 03:02:06.615628 14119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:06 dist-test-slave-f1j0 krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:06.618889 14119 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:02:06.620354 14119 webserver.cc:460] Webserver started at http://127.0.97.62:40871/ using document root <none> and password file <none>
I20260620 03:02:06.621045 14119 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:06.621143 14119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:06.621403 14119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:02:06.624490 14119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "1cac5d49f4a847c1bfbc6aa30490688f"
format_stamp: "Formatted at 2026-06-20 03:02:06 on dist-test-slave-f1j0"
server_key: "febbbba0c3fd31d6f7147ae2d06638ae"
server_key_iv: "e89c85ca18f1bdcf11ead94cd07c898b"
server_key_version: "encryptionkey@0"
I20260620 03:02:06.625242 14119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "1cac5d49f4a847c1bfbc6aa30490688f"
format_stamp: "Formatted at 2026-06-20 03:02:06 on dist-test-slave-f1j0"
server_key: "febbbba0c3fd31d6f7147ae2d06638ae"
server_key_iv: "e89c85ca18f1bdcf11ead94cd07c898b"
server_key_version: "encryptionkey@0"
I20260620 03:02:06.630836 14119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 03:02:06.634429 14134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:06.635809 14119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:02:06.635988 14119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "1cac5d49f4a847c1bfbc6aa30490688f"
format_stamp: "Formatted at 2026-06-20 03:02:06 on dist-test-slave-f1j0"
server_key: "febbbba0c3fd31d6f7147ae2d06638ae"
server_key_iv: "e89c85ca18f1bdcf11ead94cd07c898b"
server_key_version: "encryptionkey@0"
I20260620 03:02:06.636190 14119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:06.670338 14119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:06.674449 14119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:06.674773 14119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:06.693478 14119 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36753
I20260620 03:02:06.693509 14185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36753 every 8 connection(s)
I20260620 03:02:06.694928 14119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:02:06.697898 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 14119
I20260620 03:02:06.698160 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:02:06.699008 388 external_mini_cluster.cc:1428] Setting key d491918ae9d71bfcdd3e50c8fa4c1284
I20260620 03:02:06.699214 14186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:02:06.714035 14186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f: Bootstrap starting.
Jun 20 03:02:06 dist-test-slave-f1j0 krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:02:06.717423 14186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f: Neither blocks nor log segments found. Creating new log.
I20260620 03:02:06.718847 14186 log.cc:824] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:06.722306 14186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f: No bootstrap required, opened a new log
I20260620 03:02:06.726114 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:06.701798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53770 (local address 127.0.97.62:36753)
0620 03:02:06.702806 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:06.702823 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:06.703229 (+ 406us) server_negotiation.cc:407] Connection header received
0620 03:02:06.706133 (+ 2904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:06.706164 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:06.706716 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:06.707190 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:06.708500 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:06.710545 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:06.711917 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:06.712956 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:06.713141 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:06.717530 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:06.717542 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:06.717561 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:06.717634 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:06.721422 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:06.722203 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:06.722212 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:06.722221 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:06.722458 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:06.722847 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:06.722852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:06.722855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:06.723554 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:06.723848 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:06.724229 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 03:02:06.724702 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":446,"threads_started":1}
I20260620 03:02:06.730062 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:02:06.731969 14186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cac5d49f4a847c1bfbc6aa30490688f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36753 } }
I20260620 03:02:06.732429 14186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:02:06.732563 14186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cac5d49f4a847c1bfbc6aa30490688f, State: Initialized, Role: FOLLOWER
I20260620 03:02:06.733240 14186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cac5d49f4a847c1bfbc6aa30490688f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36753 } }
I20260620 03:02:06.733484 14186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:02:06.733636 14186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:02:06.733824 14186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:02:06.735986 14186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cac5d49f4a847c1bfbc6aa30490688f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36753 } }
I20260620 03:02:06.736505 14186 leader_election.cc:302] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cac5d49f4a847c1bfbc6aa30490688f; no voters:
I20260620 03:02:06.737107 14186 leader_election.cc:288] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:02:06.737251 14191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:02:06.738055 14191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f [term 1 LEADER]: Becoming Leader. State: Replica: 1cac5d49f4a847c1bfbc6aa30490688f, State: Running, Role: LEADER
I20260620 03:02:06.738566 14191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cac5d49f4a847c1bfbc6aa30490688f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36753 } }
I20260620 03:02:06.739355 14186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:02:06.741748 14192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cac5d49f4a847c1bfbc6aa30490688f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cac5d49f4a847c1bfbc6aa30490688f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36753 } } }
I20260620 03:02:06.742228 14192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f [sys.catalog]: This master's current role is: LEADER
I20260620 03:02:06.741777 14193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cac5d49f4a847c1bfbc6aa30490688f. Latest consensus state: current_term: 1 leader_uuid: "1cac5d49f4a847c1bfbc6aa30490688f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cac5d49f4a847c1bfbc6aa30490688f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36753 } } }
I20260620 03:02:06.743057 14193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f [sys.catalog]: This master's current role is: LEADER
I20260620 03:02:06.746069 14197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:02:06.751319 14197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:02:06.760166 14197 catalog_manager.cc:1347] Generated new cluster ID: 6b84ee67de5642d8b736841b95d180c5
I20260620 03:02:06.760291 14197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:02:06.806942 14197 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:02:06.808809 14197 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:02:06.821343 14197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cac5d49f4a847c1bfbc6aa30490688f: Generated new TSK 0
I20260620 03:02:06.822468 14197 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:02:06.842326 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 14119
I20260620 03:02:06.843464 14181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:06.939510 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 14119
2026-06-20T03:02:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2025 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:02:07 dist-test-slave-f1j0 krb5kdc[14214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:02:07 dist-test-slave-f1j0 krb5kdc[14214](info): set up 2 sockets
Jun 20 03:02:07 dist-test-slave-f1j0 krb5kdc[14214](info): commencing operation
krb5kdc: starting...
W20260620 03:02:08.042034 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:02:08 dist-test-slave-f1j0 krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:02:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:02:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:02:08.277146 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36449
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:02:08.605626 14227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:08.605971 14227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:08.606142 14227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:08.613453 14227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:02:08.613642 14227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:08.613744 14227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:08.613839 14227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:02:08.613911 14227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:02:08.623958 14227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:08.625409 14227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:08.626825 14227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:08.639166 14232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:08.639122 14235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:08.639092 14233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:08.640056 14227 server_base.cc:1034] running on GCE node
I20260620 03:02:08.640847 14227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:08.642373 14227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:08.643613 14227 hybrid_clock.cc:648] HybridClock initialized: now 1781924528643556 us; error 34 us; skew 500 ppm
I20260620 03:02:08.643972 14227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:08 dist-test-slave-f1j0 krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:08.647331 14227 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:02:08.648917 14227 webserver.cc:460] Webserver started at http://127.0.97.62:34355/ using document root <none> and password file <none>
I20260620 03:02:08.649660 14227 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:08.649775 14227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:08.650039 14227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:02:08.653014 14227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "c9a6bd4ede4241cf9c7581517022d598"
format_stamp: "Formatted at 2026-06-20 03:02:08 on dist-test-slave-f1j0"
server_key: "6fd6b90e8692b03cfc225221b20b865e"
server_key_iv: "d0dfd3853a4b964cf0f2e0f3b3ba1f87"
server_key_version: "encryptionkey@0"
I20260620 03:02:08.653831 14227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "c9a6bd4ede4241cf9c7581517022d598"
format_stamp: "Formatted at 2026-06-20 03:02:08 on dist-test-slave-f1j0"
server_key: "6fd6b90e8692b03cfc225221b20b865e"
server_key_iv: "d0dfd3853a4b964cf0f2e0f3b3ba1f87"
server_key_version: "encryptionkey@0"
I20260620 03:02:08.659092 14227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:02:08.662482 14242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:08.663929 14227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:02:08.664088 14227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "c9a6bd4ede4241cf9c7581517022d598"
format_stamp: "Formatted at 2026-06-20 03:02:08 on dist-test-slave-f1j0"
server_key: "6fd6b90e8692b03cfc225221b20b865e"
server_key_iv: "d0dfd3853a4b964cf0f2e0f3b3ba1f87"
server_key_version: "encryptionkey@0"
I20260620 03:02:08.664252 14227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:08.680438 14227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:08.684636 14227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:08.684891 14227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:08.703212 14227 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36449
I20260620 03:02:08.703265 14293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36449 every 8 connection(s)
I20260620 03:02:08.704665 14227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:02:08.709508 14294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:02:08.710078 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 14227
I20260620 03:02:08.710301 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:02:08.711145 388 external_mini_cluster.cc:1428] Setting key 45fc9324acb89a16d608780b9821ac74
I20260620 03:02:08.723523 14294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598: Bootstrap starting.
Jun 20 03:02:08 dist-test-slave-f1j0 krb5kdc[14214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:02:08.726696 14294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598: Neither blocks nor log segments found. Creating new log.
I20260620 03:02:08.727895 14294 log.cc:824] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:08.730944 14294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598: No bootstrap required, opened a new log
I20260620 03:02:08.733793 14297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:08.713314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.0.97.62:36449)
0620 03:02:08.714279 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:08.714291 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:08.714334 (+ 43us) server_negotiation.cc:407] Connection header received
0620 03:02:08.715936 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:08.715958 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:08.716361 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:08.716789 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:08.718384 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:08.719615 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:08.721004 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:08.721800 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:08.722001 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:08.726013 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:08.726020 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:08.726034 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:08.726073 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:08.729677 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:08.730393 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:08.730399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:08.730405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:08.730567 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:08.730949 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:08.730954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:08.730965 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:08.731517 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:08.731845 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:08.732296 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 03:02:08.732736 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":472,"threads_started":1}
I20260620 03:02:08.739500 14294 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a6bd4ede4241cf9c7581517022d598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36449 } }
I20260620 03:02:08.739861 14294 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:02:08.739981 14294 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a6bd4ede4241cf9c7581517022d598, State: Initialized, Role: FOLLOWER
I20260620 03:02:08.740552 14294 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a6bd4ede4241cf9c7581517022d598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36449 } }
I20260620 03:02:08.740777 14294 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:02:08.740926 14294 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:02:08.741078 14294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:02:08.743467 14294 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a6bd4ede4241cf9c7581517022d598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36449 } }
I20260620 03:02:08.743980 14294 leader_election.cc:302] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9a6bd4ede4241cf9c7581517022d598; no voters:
I20260620 03:02:08.744577 14294 leader_election.cc:288] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:02:08.744721 14299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:02:08.745539 14299 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 1 LEADER]: Becoming Leader. State: Replica: c9a6bd4ede4241cf9c7581517022d598, State: Running, Role: LEADER
I20260620 03:02:08.746078 14299 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a6bd4ede4241cf9c7581517022d598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36449 } }
I20260620 03:02:08.746819 14294 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:02:08.751325 14301 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9a6bd4ede4241cf9c7581517022d598. Latest consensus state: current_term: 1 leader_uuid: "c9a6bd4ede4241cf9c7581517022d598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a6bd4ede4241cf9c7581517022d598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36449 } } }
I20260620 03:02:08.751300 14300 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9a6bd4ede4241cf9c7581517022d598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a6bd4ede4241cf9c7581517022d598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36449 } } }
I20260620 03:02:08.751727 14301 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [sys.catalog]: This master's current role is: LEADER
I20260620 03:02:08.751775 14300 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [sys.catalog]: This master's current role is: LEADER
I20260620 03:02:08.758531 14311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:02:08.763036 14311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:02:08.771582 14311 catalog_manager.cc:1347] Generated new cluster ID: 7f4e1a1298ad40609d0e0e19407c5a20
I20260620 03:02:08.771700 14311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:02:08.794212 14311 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:02:08.795702 14311 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:02:08.809544 14311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598: Generated new TSK 0
I20260620 03:02:08.810513 14311 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:02:08.869408 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36449
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:02:09.208619 14320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:09.208889 14320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:09.209064 14320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:09.216315 14320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:09.216415 14320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:09.216580 14320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:02:09.226464 14320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:09.227726 14320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:09.229063 14320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:09.243028 14326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:09.243105 14328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:09.243117 14325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:09.243903 14320 server_base.cc:1034] running on GCE node
I20260620 03:02:09.244482 14320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:09.245417 14320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:09.246655 14320 hybrid_clock.cc:648] HybridClock initialized: now 1781924529246561 us; error 68 us; skew 500 ppm
I20260620 03:02:09.246932 14320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:09 dist-test-slave-f1j0 krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:09.250283 14320 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:02:09.251732 14320 webserver.cc:460] Webserver started at http://127.0.97.1:45481/ using document root <none> and password file <none>
I20260620 03:02:09.252341 14320 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:09.252439 14320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:09.252678 14320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:02:09.255538 14320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "51a4c984db1b43ddb178038c06065720"
format_stamp: "Formatted at 2026-06-20 03:02:09 on dist-test-slave-f1j0"
server_key: "398ec8d9d132bd3c2338480a6f86b874"
server_key_iv: "269704f795ae101a75b46a28812d3f69"
server_key_version: "encryptionkey@0"
I20260620 03:02:09.256240 14320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "51a4c984db1b43ddb178038c06065720"
format_stamp: "Formatted at 2026-06-20 03:02:09 on dist-test-slave-f1j0"
server_key: "398ec8d9d132bd3c2338480a6f86b874"
server_key_iv: "269704f795ae101a75b46a28812d3f69"
server_key_version: "encryptionkey@0"
I20260620 03:02:09.261113 14320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:02:09.264567 14335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:09.266072 14320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:02:09.266247 14320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "51a4c984db1b43ddb178038c06065720"
format_stamp: "Formatted at 2026-06-20 03:02:09 on dist-test-slave-f1j0"
server_key: "398ec8d9d132bd3c2338480a6f86b874"
server_key_iv: "269704f795ae101a75b46a28812d3f69"
server_key_version: "encryptionkey@0"
I20260620 03:02:09.266443 14320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:09.284323 14320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:09.288175 14320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:09.288461 14320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:09.289408 14320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:02:09.290822 14320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:02:09.290892 14320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:09.291059 14320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:02:09.291117 14320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:09.324864 14320 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33399
I20260620 03:02:09.324908 14447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33399 every 8 connection(s)
I20260620 03:02:09.326260 14320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:02:09.331584 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 14320
I20260620 03:02:09.331800 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:02:09.332485 388 external_mini_cluster.cc:1428] Setting key 13a4e2f3fb1897160912622045ac925e
Jun 20 03:02:09 dist-test-slave-f1j0 krb5kdc[14214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:02:09.349396 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:09.328824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58599 (local address 127.0.97.62:36449)
0620 03:02:09.329306 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:09.329311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:09.330339 (+ 1028us) server_negotiation.cc:407] Connection header received
0620 03:02:09.332233 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:09.332239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:09.332332 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:09.332453 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:09.334392 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:09.335204 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:09.337825 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:09.338257 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:09.338361 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:09.342704 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:09.342709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:09.342712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:09.342743 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:09.345856 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:09.346800 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:09.346804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:09.346806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:09.346980 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:09.347489 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:09.347494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:09.347497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:09.347805 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:09.347924 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:09.348847 (+ 923us) server_negotiation.cc:299] Negotiation successful
0620 03:02:09.349053 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":271,"threads_started":1}
I20260620 03:02:09.350991 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:09.329447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:58599)
0620 03:02:09.330186 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:09.330222 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:09.331968 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:09.332770 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:09.332782 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:09.333517 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:09.334192 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:09.334237 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:09.335375 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:09.335383 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:09.337650 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:09.337668 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:09.338496 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:09.338506 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:09.338852 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:09.339627 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:09.339662 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:09.342440 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:09.346004 (+ 3564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:09.346016 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:09.346035 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:09.346684 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:09.347092 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:09.347099 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:09.347106 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:09.347357 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:09.347936 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:09.347946 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:09.348423 (+ 477us) client_negotiation.cc:769] Sending connection context
0620 03:02:09.348755 (+ 332us) client_negotiation.cc:240] Negotiation successful
0620 03:02:09.349390 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":328,"threads_started":1}
I20260620 03:02:09.352687 14448 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36449
I20260620 03:02:09.352995 14448 heartbeater.cc:464] Registering TS with master...
I20260620 03:02:09.353878 14448 heartbeater.cc:511] Master 127.0.97.62:36449 requested a full tablet report, sending...
I20260620 03:02:09.356112 14259 ts_manager.cc:194] Registered new tserver with Master: 51a4c984db1b43ddb178038c06065720 (127.0.97.1:33399)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 03:02:09.357722 14259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:02:09.386883 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36449
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:02:09.715340 14454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:09.715657 14454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:09.715812 14454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:09.723157 14454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:09.723308 14454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:09.723489 14454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:02:09.733215 14454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:09.734584 14454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:09.735886 14454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:09.749475 14462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:09.749433 14460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:09.749532 14459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:09.750288 14454 server_base.cc:1034] running on GCE node
I20260620 03:02:09.750949 14454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:09.751897 14454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:09.753175 14454 hybrid_clock.cc:648] HybridClock initialized: now 1781924529753101 us; error 54 us; skew 500 ppm
I20260620 03:02:09.753562 14454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:09 dist-test-slave-f1j0 krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:09.756642 14454 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:02:09.758090 14454 webserver.cc:460] Webserver started at http://127.0.97.2:34377/ using document root <none> and password file <none>
I20260620 03:02:09.758767 14454 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:09.758911 14454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:09.759218 14454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:02:09.762176 14454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "eb7608719a3c490891988f688bd61042"
format_stamp: "Formatted at 2026-06-20 03:02:09 on dist-test-slave-f1j0"
server_key: "de6a0f6520974a81ffddbf414192f3bc"
server_key_iv: "4d4e3829ef5cda654befc53562c82739"
server_key_version: "encryptionkey@0"
I20260620 03:02:09.762933 14454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "eb7608719a3c490891988f688bd61042"
format_stamp: "Formatted at 2026-06-20 03:02:09 on dist-test-slave-f1j0"
server_key: "de6a0f6520974a81ffddbf414192f3bc"
server_key_iv: "4d4e3829ef5cda654befc53562c82739"
server_key_version: "encryptionkey@0"
I20260620 03:02:09.768285 14454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:02:09.772012 14469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:09.773324 14454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 03:02:09.773496 14454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "eb7608719a3c490891988f688bd61042"
format_stamp: "Formatted at 2026-06-20 03:02:09 on dist-test-slave-f1j0"
server_key: "de6a0f6520974a81ffddbf414192f3bc"
server_key_iv: "4d4e3829ef5cda654befc53562c82739"
server_key_version: "encryptionkey@0"
I20260620 03:02:09.773762 14454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:09.801038 14454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:09.805037 14454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:09.805352 14454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:09.806355 14454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:02:09.807659 14454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:02:09.807762 14454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:09.807912 14454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:02:09.807984 14454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:09.840526 14454 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42605
I20260620 03:02:09.840610 14581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42605 every 8 connection(s)
I20260620 03:02:09.841845 14454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:02:09.849478 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 14454
I20260620 03:02:09.849697 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:02:09.850178 388 external_mini_cluster.cc:1428] Setting key f440254f0abd60abd5f7956b6bb8d996
Jun 20 03:02:09 dist-test-slave-f1j0 krb5kdc[14214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:02:09.863651 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:09.844406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48831 (local address 127.0.97.62:36449)
0620 03:02:09.844624 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:09.844630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:09.845894 (+ 1264us) server_negotiation.cc:407] Connection header received
0620 03:02:09.847827 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:09.847833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:09.847920 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:09.848009 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:09.849958 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:09.850891 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:09.852763 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:09.853171 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:09.853294 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:09.857758 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:09.857763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:09.857766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:09.857804 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:09.860423 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:09.861229 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:09.861233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:09.861235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:09.861360 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:09.861874 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:09.861885 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:09.861887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:09.862147 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:09.862326 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:09.863040 (+ 714us) server_negotiation.cc:299] Negotiation successful
0620 03:02:09.863302 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 03:02:09.864667 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:09.844910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:48831)
0620 03:02:09.845731 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:09.845770 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:09.847510 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:09.848263 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:09.848271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:09.848796 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:09.849705 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:09.849734 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:09.851050 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:09.851057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:09.852599 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:09.852613 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:09.853422 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:09.853430 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:09.853799 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:09.854579 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:09.854622 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:09.857474 (+ 2852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:09.860595 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:09.860605 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:09.860623 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:09.861078 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:09.861477 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:09.861482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:09.861486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:09.861752 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:09.862293 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:09.862303 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:09.862662 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 03:02:09.862941 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 03:02:09.863498 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":305,"threads_started":1}
I20260620 03:02:09.866215 14582 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36449
I20260620 03:02:09.866542 14582 heartbeater.cc:464] Registering TS with master...
I20260620 03:02:09.867296 14582 heartbeater.cc:511] Master 127.0.97.62:36449 requested a full tablet report, sending...
I20260620 03:02:09.869143 14259 ts_manager.cc:194] Registered new tserver with Master: eb7608719a3c490891988f688bd61042 (127.0.97.2:42605)
I20260620 03:02:09.870283 14259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48831
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:02:09.904721 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36449
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:02:10.231738 14587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:10.232005 14587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:10.232100 14587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:10.239298 14587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:10.239406 14587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:10.239518 14587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:02:10.249311 14587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:10.250710 14587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:10.252086 14587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:10.265226 14595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:10.265297 14593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:10.265343 14592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:10.266014 14587 server_base.cc:1034] running on GCE node
I20260620 03:02:10.266564 14587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:10.267508 14587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:10.268764 14587 hybrid_clock.cc:648] HybridClock initialized: now 1781924530268686 us; error 56 us; skew 500 ppm
I20260620 03:02:10.269047 14587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:10 dist-test-slave-f1j0 krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:10.272374 14587 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:02:10.274024 14587 webserver.cc:460] Webserver started at http://127.0.97.3:44829/ using document root <none> and password file <none>
I20260620 03:02:10.274729 14587 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:10.274863 14587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:10.275171 14587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:02:10.278301 14587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "72aa7ecd9ca24eacb603857634cb630e"
format_stamp: "Formatted at 2026-06-20 03:02:10 on dist-test-slave-f1j0"
server_key: "c104300cb7528619f36c7e062f9b08e6"
server_key_iv: "da772be2274f2c3b3572c1cca3722177"
server_key_version: "encryptionkey@0"
I20260620 03:02:10.279064 14587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "72aa7ecd9ca24eacb603857634cb630e"
format_stamp: "Formatted at 2026-06-20 03:02:10 on dist-test-slave-f1j0"
server_key: "c104300cb7528619f36c7e062f9b08e6"
server_key_iv: "da772be2274f2c3b3572c1cca3722177"
server_key_version: "encryptionkey@0"
I20260620 03:02:10.284310 14587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 03:02:10.287864 14602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:10.289237 14587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:02:10.289407 14587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "72aa7ecd9ca24eacb603857634cb630e"
format_stamp: "Formatted at 2026-06-20 03:02:10 on dist-test-slave-f1j0"
server_key: "c104300cb7528619f36c7e062f9b08e6"
server_key_iv: "da772be2274f2c3b3572c1cca3722177"
server_key_version: "encryptionkey@0"
I20260620 03:02:10.289636 14587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:10.308997 14587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:10.312968 14587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:10.313268 14587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:10.314296 14587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:02:10.315598 14587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:02:10.315668 14587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:10.315830 14587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:02:10.315882 14587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:10.349622 14587 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35411
I20260620 03:02:10.349643 14714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35411 every 8 connection(s)
I20260620 03:02:10.351014 14587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:02:10.357159 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 14587
I20260620 03:02:10.357371 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:02:10.357960 388 external_mini_cluster.cc:1428] Setting key eb2e1a269d78ac33d946542c05b122cc
I20260620 03:02:10.361369 14448 heartbeater.cc:503] Master 127.0.97.62:36449 was elected leader, sending a full tablet report...
Jun 20 03:02:10 dist-test-slave-f1j0 krb5kdc[14214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:02:10.371579 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:10.353505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56275 (local address 127.0.97.62:36449)
0620 03:02:10.353734 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:10.353738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:10.355022 (+ 1284us) server_negotiation.cc:407] Connection header received
0620 03:02:10.357456 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:10.357462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:10.357559 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:10.357697 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:10.359216 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.359995 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.361762 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.362282 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.362426 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:10.365936 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:10.365941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:10.365944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:10.365984 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:10.368568 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:10.369312 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:10.369316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:10.369319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:10.369440 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:10.369885 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:10.369888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:10.369890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:10.370138 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:10.370299 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:10.371017 (+ 718us) server_negotiation.cc:299] Negotiation successful
0620 03:02:10.371234 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 03:02:10.372722 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:10.354069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:56275)
0620 03:02:10.354845 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:10.354894 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:10.357135 (+ 2241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:10.357884 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:10.357892 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:10.358466 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:10.359036 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:10.359054 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.360163 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.360168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:10.361636 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:10.361650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.362520 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.362528 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:10.362767 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:10.363448 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:10.363474 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:10.365743 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:10.368706 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:10.368713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:10.368726 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:10.369200 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:10.369529 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:10.369532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:10.369534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:10.369767 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:10.370308 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:10.370319 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:10.370645 (+ 326us) client_negotiation.cc:769] Sending connection context
0620 03:02:10.370909 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 03:02:10.371442 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":344,"threads_started":1}
I20260620 03:02:10.374094 14715 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36449
I20260620 03:02:10.374387 14715 heartbeater.cc:464] Registering TS with master...
I20260620 03:02:10.375082 14715 heartbeater.cc:511] Master 127.0.97.62:36449 requested a full tablet report, sending...
I20260620 03:02:10.376392 14259 ts_manager.cc:194] Registered new tserver with Master: 72aa7ecd9ca24eacb603857634cb630e (127.0.97.3:35411)
I20260620 03:02:10.377259 14259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56275
I20260620 03:02:10.385466 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:02:10.402721 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:10.390060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.0.97.62:36449)
0620 03:02:10.390301 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:10.390310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:10.390654 (+ 344us) server_negotiation.cc:407] Connection header received
0620 03:02:10.390810 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:10.390814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:10.390879 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:10.390965 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:10.392262 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.393158 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.394619 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.395039 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.395143 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:10.397518 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:10.397523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:10.397526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:10.397560 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:10.400136 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:10.400783 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:10.400787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:10.400790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:10.400912 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:10.401251 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:10.401254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:10.401257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:10.401702 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:10.401899 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:10.402196 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 03:02:10.402359 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 03:02:10.406836 14259 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:02:10.414381 14259 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:02:10.447165 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:10.435885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.0.97.1:33399)
0620 03:02:10.437283 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:10.437296 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:10.438793 (+ 1497us) server_negotiation.cc:407] Connection header received
0620 03:02:10.439146 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:10.439154 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:10.439358 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:10.439573 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:10.440695 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.442000 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.444502 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.445892 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.446022 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:10.446265 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:10.446597 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 03:02:10.446792 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":434,"threads_started":1}
I20260620 03:02:10.447575 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:10.426685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42605 (local address 127.0.0.1:45980)
0620 03:02:10.433296 (+ 6611us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:10.433357 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:10.433595 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:10.439182 (+ 5587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:10.439189 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:10.439222 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:10.439757 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:10.439779 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.441690 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.441699 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:10.444745 (+ 3046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:10.444761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.446559 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.446565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:10.446752 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:10.446776 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 03:02:10.446891 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 03:02:10.447082 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6229,"mutex_wait_us":242,"thread_start_us":6063,"threads_started":1}
I20260620 03:02:10.447934 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:10.427458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45980 (local address 127.0.97.2:42605)
0620 03:02:10.438570 (+ 11112us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:10.438578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:10.438606 (+ 28us) server_negotiation.cc:407] Connection header received
0620 03:02:10.438697 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:10.438714 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:10.438951 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:10.439190 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:10.439993 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.441450 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.444916 (+ 3466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.446373 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.446513 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:10.446695 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:10.446926 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 03:02:10.447086 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10930,"thread_start_us":10792,"threads_started":1}
I20260620 03:02:10.448096 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:10.436749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:43190)
0620 03:02:10.438679 (+ 1930us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:10.438697 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:10.438907 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:10.439827 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:10.439832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:10.439862 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:10.440489 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:10.440501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.442141 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.442146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:10.444341 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:10.444353 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.446094 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.446100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:10.446307 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:10.446360 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 03:02:10.447478 (+ 1118us) client_negotiation.cc:240] Negotiation successful
0620 03:02:10.447632 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1785,"mutex_wait_us":22,"thread_start_us":722,"threads_started":1}
I20260620 03:02:10.448405 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:10.427019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35411 (local address 127.0.0.1:34910)
0620 03:02:10.435275 (+ 8256us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:10.435298 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:10.435438 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:10.439052 (+ 3614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:10.439060 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:10.439097 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:10.439823 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:10.439840 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.441634 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.441643 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:10.444745 (+ 3102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:10.444761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.447487 (+ 2726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.447494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:10.447687 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:10.447722 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 03:02:10.447803 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 03:02:10.447933 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8095,"mutex_wait_us":211,"thread_start_us":7944,"threads_started":1}
I20260620 03:02:10.452445 14517 tablet_service.cc:1467] Processing CreateTablet for tablet 4937e3b1a56a4eeb87686ffffa9972f8 (DEFAULT_TABLE table=test-table [id=96aeda512a2f4eaa9b5e29dfeb8b91da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:02:10.452445 14383 tablet_service.cc:1467] Processing CreateTablet for tablet 4937e3b1a56a4eeb87686ffffa9972f8 (DEFAULT_TABLE table=test-table [id=96aeda512a2f4eaa9b5e29dfeb8b91da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:02:10.453760 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:10.427537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34910 (local address 127.0.97.3:35411)
0620 03:02:10.438326 (+ 10789us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:10.438335 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:10.438371 (+ 36us) server_negotiation.cc:407] Connection header received
0620 03:02:10.438482 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:10.438494 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:10.438798 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:10.439513 (+ 715us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:10.440088 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.441454 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.445707 (+ 4253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.447162 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.448786 (+ 1624us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:10.449031 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:10.453318 (+ 4287us) server_negotiation.cc:299] Negotiation successful
0620 03:02:10.453456 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10558,"thread_start_us":10372,"threads_started":1}
I20260620 03:02:10.454066 14383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4937e3b1a56a4eeb87686ffffa9972f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:02:10.454068 14517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4937e3b1a56a4eeb87686ffffa9972f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:02:10.457010 14650 tablet_service.cc:1467] Processing CreateTablet for tablet 4937e3b1a56a4eeb87686ffffa9972f8 (DEFAULT_TABLE table=test-table [id=96aeda512a2f4eaa9b5e29dfeb8b91da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:02:10.458174 14650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4937e3b1a56a4eeb87686ffffa9972f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:02:10.469440 14733 tablet_bootstrap.cc:492] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e: Bootstrap starting.
I20260620 03:02:10.472736 14733 tablet_bootstrap.cc:654] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e: Neither blocks nor log segments found. Creating new log.
I20260620 03:02:10.473655 14735 tablet_bootstrap.cc:492] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042: Bootstrap starting.
I20260620 03:02:10.474303 14733 log.cc:824] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:10.477015 14734 tablet_bootstrap.cc:492] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720: Bootstrap starting.
I20260620 03:02:10.477771 14735 tablet_bootstrap.cc:654] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042: Neither blocks nor log segments found. Creating new log.
I20260620 03:02:10.478303 14733 tablet_bootstrap.cc:492] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e: No bootstrap required, opened a new log
I20260620 03:02:10.478578 14733 ts_tablet_manager.cc:1397] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 03:02:10.479136 14735 log.cc:824] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:10.480355 14734 tablet_bootstrap.cc:654] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720: Neither blocks nor log segments found. Creating new log.
I20260620 03:02:10.481304 14734 log.cc:824] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:10.482328 14735 tablet_bootstrap.cc:492] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042: No bootstrap required, opened a new log
I20260620 03:02:10.482604 14735 ts_tablet_manager.cc:1397] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260620 03:02:10.483851 14734 tablet_bootstrap.cc:492] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720: No bootstrap required, opened a new log
I20260620 03:02:10.484109 14734 ts_tablet_manager.cc:1397] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 03:02:10.490398 14733 raft_consensus.cc:348] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } } peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } }
I20260620 03:02:10.490931 14733 raft_consensus.cc:374] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:02:10.491041 14733 raft_consensus.cc:729] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72aa7ecd9ca24eacb603857634cb630e, State: Initialized, Role: FOLLOWER
I20260620 03:02:10.491734 14733 consensus_queue.cc:260] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } } peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } }
I20260620 03:02:10.492100 14734 raft_consensus.cc:348] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } } peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } }
I20260620 03:02:10.492563 14734 raft_consensus.cc:374] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:02:10.492671 14734 raft_consensus.cc:729] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a4c984db1b43ddb178038c06065720, State: Initialized, Role: FOLLOWER
I20260620 03:02:10.493242 14715 heartbeater.cc:503] Master 127.0.97.62:36449 was elected leader, sending a full tablet report...
I20260620 03:02:10.493753 14733 ts_tablet_manager.cc:1428] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260620 03:02:10.494457 14734 consensus_queue.cc:260] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } } peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } }
I20260620 03:02:10.494376 14735 raft_consensus.cc:348] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } } peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } }
I20260620 03:02:10.494848 14735 raft_consensus.cc:374] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:02:10.494976 14735 raft_consensus.cc:729] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7608719a3c490891988f688bd61042, State: Initialized, Role: FOLLOWER
I20260620 03:02:10.495654 14735 consensus_queue.cc:260] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } } peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } }
I20260620 03:02:10.496536 14734 ts_tablet_manager.cc:1428] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 03:02:10.497712 14582 heartbeater.cc:503] Master 127.0.97.62:36449 was elected leader, sending a full tablet report...
I20260620 03:02:10.498304 14735 ts_tablet_manager.cc:1428] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260620 03:02:10.553776 14740 raft_consensus.cc:482] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:02:10.554076 14740 raft_consensus.cc:504] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } } peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } }
I20260620 03:02:10.555845 14740 leader_election.cc:288] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb7608719a3c490891988f688bd61042 (127.0.97.2:42605), 72aa7ecd9ca24eacb603857634cb630e (127.0.97.3:35411)
I20260620 03:02:10.564785 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:10.556552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56693 (local address 127.0.97.3:35411)
0620 03:02:10.556822 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:10.556831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:10.558033 (+ 1202us) server_negotiation.cc:407] Connection header received
0620 03:02:10.558175 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:10.558180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:10.558251 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:10.558319 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:10.558906 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.559934 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.562463 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.563675 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.563818 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:10.563919 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:10.564390 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 03:02:10.564520 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 03:02:10.564786 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:10.556357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42801 (local address 127.0.97.2:42605)
0620 03:02:10.557707 (+ 1350us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:10.557715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:10.557735 (+ 20us) server_negotiation.cc:407] Connection header received
0620 03:02:10.557813 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:10.557817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:10.557906 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:10.558010 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:10.558623 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.559716 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.562466 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.563693 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.563818 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:10.563917 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:10.564390 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 03:02:10.564520 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 03:02:10.565019 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:10.556333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42605 (local address 127.0.97.1:42801)
0620 03:02:10.557300 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:10.557319 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:10.557426 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:10.558042 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:10.558047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:10.558068 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:10.558465 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:10.558476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.559845 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.559849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:10.562305 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:10.562319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.563986 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.563990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:10.564175 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:10.564250 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 03:02:10.564347 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 03:02:10.564530 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":139,"thread_start_us":1343,"threads_started":1}
I20260620 03:02:10.565045 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:10.556440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35411 (local address 127.0.97.1:56693)
0620 03:02:10.557911 (+ 1471us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:10.557924 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:10.558051 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:10.558373 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:10.558380 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:10.558411 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:10.558752 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:10.558760 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.560054 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.560059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:10.562305 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:10.562319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.563969 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.563977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:10.564238 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:10.564262 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 03:02:10.564347 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 03:02:10.564530 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1384,"mutex_wait_us":93,"thread_start_us":505,"threads_started":1}
I20260620 03:02:10.566188 14670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4937e3b1a56a4eeb87686ffffa9972f8" candidate_uuid: "51a4c984db1b43ddb178038c06065720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72aa7ecd9ca24eacb603857634cb630e" is_pre_election: true
I20260620 03:02:10.566246 14537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4937e3b1a56a4eeb87686ffffa9972f8" candidate_uuid: "51a4c984db1b43ddb178038c06065720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb7608719a3c490891988f688bd61042" is_pre_election: true
I20260620 03:02:10.566792 14670 raft_consensus.cc:2454] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51a4c984db1b43ddb178038c06065720 in term 0.
I20260620 03:02:10.566800 14537 raft_consensus.cc:2454] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51a4c984db1b43ddb178038c06065720 in term 0.
I20260620 03:02:10.567620 14338 leader_election.cc:302] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 51a4c984db1b43ddb178038c06065720, 72aa7ecd9ca24eacb603857634cb630e, eb7608719a3c490891988f688bd61042; no voters:
I20260620 03:02:10.568145 14740 raft_consensus.cc:2781] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:02:10.568303 14740 raft_consensus.cc:482] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:02:10.568408 14740 raft_consensus.cc:3037] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:02:10.570818 14740 raft_consensus.cc:504] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } } peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } }
I20260620 03:02:10.571365 14740 leader_election.cc:288] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 1 election: Requested vote from peers eb7608719a3c490891988f688bd61042 (127.0.97.2:42605), 72aa7ecd9ca24eacb603857634cb630e (127.0.97.3:35411)
I20260620 03:02:10.571768 14537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4937e3b1a56a4eeb87686ffffa9972f8" candidate_uuid: "51a4c984db1b43ddb178038c06065720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb7608719a3c490891988f688bd61042"
I20260620 03:02:10.571815 14670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4937e3b1a56a4eeb87686ffffa9972f8" candidate_uuid: "51a4c984db1b43ddb178038c06065720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72aa7ecd9ca24eacb603857634cb630e"
I20260620 03:02:10.571956 14537 raft_consensus.cc:3037] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:02:10.572027 14670 raft_consensus.cc:3037] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:02:10.574445 14537 raft_consensus.cc:2454] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a4c984db1b43ddb178038c06065720 in term 1.
I20260620 03:02:10.574509 14670 raft_consensus.cc:2454] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a4c984db1b43ddb178038c06065720 in term 1.
I20260620 03:02:10.574848 14337 leader_election.cc:302] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a4c984db1b43ddb178038c06065720, eb7608719a3c490891988f688bd61042; no voters:
I20260620 03:02:10.575145 14740 raft_consensus.cc:2781] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:02:10.575569 14740 raft_consensus.cc:686] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [term 1 LEADER]: Becoming Leader. State: Replica: 51a4c984db1b43ddb178038c06065720, State: Running, Role: LEADER
I20260620 03:02:10.576061 14740 consensus_queue.cc:237] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } } peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } }
W20260620 03:02:10.579977 14449 tablet.cc:2369] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:02:10.580963 14257 catalog_manager.cc:5495] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51a4c984db1b43ddb178038c06065720 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "51a4c984db1b43ddb178038c06065720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } health_report { overall_health: HEALTHY } } }
W20260620 03:02:10.593945 14583 tablet.cc:2369] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:02:10.602557 14716 tablet.cc:2369] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:02:10.629165 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:10.623501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.0.97.1:33399)
0620 03:02:10.623825 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:10.623832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:10.623851 (+ 19us) server_negotiation.cc:407] Connection header received
0620 03:02:10.624134 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:10.624139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:10.624238 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:10.624397 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:02:10.624780 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.625522 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.626967 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.627388 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.627527 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:10.628057 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:02:10.628571 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:02:10.628738 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:10.628795 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 03:02:10.628876 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260620 03:02:10.639860 14670 raft_consensus.cc:1261] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e [term 1 FOLLOWER]: Refusing update from remote peer 51a4c984db1b43ddb178038c06065720: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:02:10.639926 14537 raft_consensus.cc:1261] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042 [term 1 FOLLOWER]: Refusing update from remote peer 51a4c984db1b43ddb178038c06065720: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:02:10.641099 14740 consensus_queue.cc:1035] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:02:10.641746 14744 consensus_queue.cc:1035] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 03:02:10.659355 14747 mvcc.cc:205] Tried to move back new op lower bound from 7298762877488533504 to 7298762877243633664. Current Snapshot: MvccSnapshot[applied={T|T < 7298762877488533504}]
I20260620 03:02:10.676000 14259 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:02:10.676342 14259 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:02:10.681945 14259 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598: Sending DeleteTablet for 3 replicas of tablet 4937e3b1a56a4eeb87686ffffa9972f8
I20260620 03:02:10.683218 14517 tablet_service.cc:1514] Processing DeleteTablet for tablet 4937e3b1a56a4eeb87686ffffa9972f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:02:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45980
I20260620 03:02:10.684561 14650 tablet_service.cc:1514] Processing DeleteTablet for tablet 4937e3b1a56a4eeb87686ffffa9972f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:02:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34910
I20260620 03:02:10.689971 14383 tablet_service.cc:1514] Processing DeleteTablet for tablet 4937e3b1a56a4eeb87686ffffa9972f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:02:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43190
I20260620 03:02:10.691275 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 14320
I20260620 03:02:10.694037 14756 tablet_replica.cc:330] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042: stopping tablet replica
I20260620 03:02:10.694638 14756 raft_consensus.cc:2229] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:02:10.695281 14756 raft_consensus.cc:2258] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:02:10.697682 14756 ts_tablet_manager.cc:1905] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:02:10.705956 14757 tablet_replica.cc:330] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e: stopping tablet replica
I20260620 03:02:10.706398 14757 raft_consensus.cc:2229] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:02:10.706506 14758 tablet_replica.cc:330] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720: stopping tablet replica
I20260620 03:02:10.706921 14757 raft_consensus.cc:2258] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:02:10.707161 14758 raft_consensus.cc:2229] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:02:10.707901 14758 raft_consensus.cc:2258] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:02:10.710278 14758 ts_tablet_manager.cc:1905] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 03:02:10 dist-test-slave-f1j0 krb5kdc[14214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:02:10.717911 14758 ts_tablet_manager.cc:1918] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:02:10.718092 14758 log.cc:1196] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/4937e3b1a56a4eeb87686ffffa9972f8
I20260620 03:02:10.718677 14758 ts_tablet_manager.cc:1939] T 4937e3b1a56a4eeb87686ffffa9972f8 P 51a4c984db1b43ddb178038c06065720: Deleting consensus metadata
I20260620 03:02:10.720602 14245 catalog_manager.cc:4841] TS 51a4c984db1b43ddb178038c06065720 (127.0.97.1:33399): tablet 4937e3b1a56a4eeb87686ffffa9972f8 (table test-table [id=96aeda512a2f4eaa9b5e29dfeb8b91da]) successfully deleted
I20260620 03:02:10.721738 14756 ts_tablet_manager.cc:1918] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:02:10.721874 14756 log.cc:1196] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/4937e3b1a56a4eeb87686ffffa9972f8
I20260620 03:02:10.722294 14756 ts_tablet_manager.cc:1939] T 4937e3b1a56a4eeb87686ffffa9972f8 P eb7608719a3c490891988f688bd61042: Deleting consensus metadata
I20260620 03:02:10.723912 14244 catalog_manager.cc:4841] TS eb7608719a3c490891988f688bd61042 (127.0.97.2:42605): tablet 4937e3b1a56a4eeb87686ffffa9972f8 (table test-table [id=96aeda512a2f4eaa9b5e29dfeb8b91da]) successfully deleted
I20260620 03:02:10.725553 14757 ts_tablet_manager.cc:1905] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:02:10.726281 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:10.692568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43204 (local address 127.0.97.1:33399)
0620 03:02:10.692796 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:10.692802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:10.706138 (+ 13336us) server_negotiation.cc:407] Connection header received
0620 03:02:10.706508 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:10.706514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:10.706598 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:10.706680 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:10.708823 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.709969 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.712005 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.712741 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.712846 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:10.718455 (+ 5609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:10.718471 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:10.718492 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:10.718543 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:10.722869 (+ 4326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:10.723768 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:10.723779 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:10.723784 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:10.723975 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:10.724429 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:10.724435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:10.724438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:10.724894 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:10.725218 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:10.725561 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 03:02:10.725912 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 03:02:10.727236 14443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:10.733803 14757 ts_tablet_manager.cc:1918] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:02:10.734016 14757 log.cc:1196] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/4937e3b1a56a4eeb87686ffffa9972f8
I20260620 03:02:10.734558 14757 ts_tablet_manager.cc:1939] T 4937e3b1a56a4eeb87686ffffa9972f8 P 72aa7ecd9ca24eacb603857634cb630e: Deleting consensus metadata
I20260620 03:02:10.736310 14245 catalog_manager.cc:4841] TS 72aa7ecd9ca24eacb603857634cb630e (127.0.97.3:35411): tablet 4937e3b1a56a4eeb87686ffffa9972f8 (table test-table [id=96aeda512a2f4eaa9b5e29dfeb8b91da]) successfully deleted
I20260620 03:02:10.827912 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 14320
I20260620 03:02:10.845974 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 14454
Jun 20 03:02:10 dist-test-slave-f1j0 krb5kdc[14214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:02:10.863027 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:10.846744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45994 (local address 127.0.97.2:42605)
0620 03:02:10.846907 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:10.846912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:10.846930 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:02:10.847126 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:10.847132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:10.847228 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:10.847324 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:10.848635 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.849917 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.851353 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.851995 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.852112 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:10.856421 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:10.856434 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:10.856451 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:10.856491 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:10.860328 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:10.861117 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:10.861126 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:10.861130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:10.861283 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:10.861622 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:10.861627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:10.861629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:10.862101 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:10.862323 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:10.862581 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 03:02:10.862762 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 03:02:10.863687 14577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:10.954658 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 14454
I20260620 03:02:10.971853 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 14587
Jun 20 03:02:10 dist-test-slave-f1j0 krb5kdc[14214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:02:10.987139 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:10.972629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.0.97.3:35411)
0620 03:02:10.972823 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:10.972830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:10.972898 (+ 68us) server_negotiation.cc:407] Connection header received
0620 03:02:10.973092 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:10.973097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:10.973176 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:10.973283 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:10.974667 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.975400 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.976708 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:10.977099 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:10.977233 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:10.981500 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:10.981515 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:10.981530 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:10.981609 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:10.984155 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:10.984822 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:10.984838 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:10.984840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:10.984970 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:10.985329 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:10.985335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:10.985339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:10.985917 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:10.986218 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:10.986576 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 03:02:10.986817 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 03:02:10.987849 14710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:11.075954 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 14587
I20260620 03:02:11.092603 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 14227
I20260620 03:02:11.093796 14289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:11.205184 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 14227
I20260620 03:02:11.226394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36449
--webserver_interface=127.0.97.62
--webserver_port=34355
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:02:11.554203 14764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:11.554489 14764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:11.554651 14764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:11.561998 14764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:02:11.562115 14764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:11.562233 14764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:11.562316 14764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:02:11.562374 14764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:02:11.572678 14764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:11.574098 14764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:11.575515 14764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:11.599723 14773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:11.599810 14771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:11.600255 14770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:11.600939 14764 server_base.cc:1034] running on GCE node
I20260620 03:02:11.602015 14764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:11.603825 14764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:11.605077 14764 hybrid_clock.cc:648] HybridClock initialized: now 1781924531605020 us; error 54 us; skew 500 ppm
I20260620 03:02:11.605511 14764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:11 dist-test-slave-f1j0 krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:11.609359 14764 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:02:11.611310 14764 webserver.cc:460] Webserver started at http://127.0.97.62:34355/ using document root <none> and password file <none>
I20260620 03:02:11.612107 14764 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:11.612298 14764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:11.618731 14764 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 03:02:11.622044 14780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:11.623569 14764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:02:11.623786 14764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "c9a6bd4ede4241cf9c7581517022d598"
format_stamp: "Formatted at 2026-06-20 03:02:08 on dist-test-slave-f1j0"
server_key: "6fd6b90e8692b03cfc225221b20b865e"
server_key_iv: "d0dfd3853a4b964cf0f2e0f3b3ba1f87"
server_key_version: "encryptionkey@0"
I20260620 03:02:11.624934 14764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:11.641897 14764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:11.646044 14764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:11.646353 14764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:11.665375 14764 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36449
I20260620 03:02:11.665385 14831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36449 every 8 connection(s)
I20260620 03:02:11.666997 14764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:02:11.669016 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 14764
I20260620 03:02:11.669879 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33399
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:36449
--webserver_port=45481
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:02:11.674405 14832 sys_catalog.cc:263] Verifying existing consensus state
I20260620 03:02:11.677855 14832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598: Bootstrap starting.
I20260620 03:02:11.697424 14832 log.cc:824] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:11.709126 14832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 03:02:11.709782 14832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598: Bootstrap complete.
I20260620 03:02:11.720583 14832 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a6bd4ede4241cf9c7581517022d598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36449 } }
I20260620 03:02:11.721447 14832 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a6bd4ede4241cf9c7581517022d598, State: Initialized, Role: FOLLOWER
I20260620 03:02:11.722086 14832 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a6bd4ede4241cf9c7581517022d598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36449 } }
I20260620 03:02:11.722323 14832 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:02:11.722435 14832 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:02:11.722617 14832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 1 FOLLOWER]: Advancing to term 2
I20260620 03:02:11.726467 14832 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a6bd4ede4241cf9c7581517022d598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36449 } }
I20260620 03:02:11.726998 14832 leader_election.cc:302] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9a6bd4ede4241cf9c7581517022d598; no voters:
I20260620 03:02:11.727584 14832 leader_election.cc:288] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 03:02:11.727777 14836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 03:02:11.728884 14836 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 2 LEADER]: Becoming Leader. State: Replica: c9a6bd4ede4241cf9c7581517022d598, State: Running, Role: LEADER
I20260620 03:02:11.729557 14836 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a6bd4ede4241cf9c7581517022d598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36449 } }
I20260620 03:02:11.729872 14832 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:02:11.732010 14837 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c9a6bd4ede4241cf9c7581517022d598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a6bd4ede4241cf9c7581517022d598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36449 } } }
I20260620 03:02:11.732374 14837 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [sys.catalog]: This master's current role is: LEADER
I20260620 03:02:11.734205 14838 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9a6bd4ede4241cf9c7581517022d598. Latest consensus state: current_term: 2 leader_uuid: "c9a6bd4ede4241cf9c7581517022d598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a6bd4ede4241cf9c7581517022d598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36449 } } }
I20260620 03:02:11.734498 14838 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [sys.catalog]: This master's current role is: LEADER
I20260620 03:02:11.734871 14841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:02:11.744242 14841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:02:11.746023 14841 catalog_manager.cc:1259] Loaded cluster ID: 7f4e1a1298ad40609d0e0e19407c5a20
I20260620 03:02:11.746140 14841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:02:11.750335 14841 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 03:02:11.768491 14841 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 03:02:11.770152 14853 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 03:02:11.778271 14841 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:02:11.779955 14841 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598: Loaded TSK: 0
I20260620 03:02:11.781409 14841 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 03:02:12.015813 14834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:12.016146 14834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:12.016314 14834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:12.023695 14834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:12.023856 14834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:12.024030 14834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:02:12.034349 14834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45481
--tserver_master_addrs=127.0.97.62:36449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:12.035760 14834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:12.037204 14834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:12.052098 14859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:12.052253 14860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:12.052538 14862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:12.054157 14834 server_base.cc:1034] running on GCE node
I20260620 03:02:12.054854 14834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:12.055836 14834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:12.057067 14834 hybrid_clock.cc:648] HybridClock initialized: now 1781924532057003 us; error 58 us; skew 500 ppm
I20260620 03:02:12.057389 14834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:12 dist-test-slave-f1j0 krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:12.063637 14834 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:02:12.065338 14834 webserver.cc:460] Webserver started at http://127.0.97.1:45481/ using document root <none> and password file <none>
I20260620 03:02:12.066128 14834 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:12.066291 14834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:12.071894 14834 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 03:02:12.074788 14869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:12.076318 14834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:02:12.076490 14834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "51a4c984db1b43ddb178038c06065720"
format_stamp: "Formatted at 2026-06-20 03:02:09 on dist-test-slave-f1j0"
server_key: "398ec8d9d132bd3c2338480a6f86b874"
server_key_iv: "269704f795ae101a75b46a28812d3f69"
server_key_version: "encryptionkey@0"
I20260620 03:02:12.077543 14834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:12.120738 14834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:12.125403 14834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:12.125730 14834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:12.126734 14834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:02:12.128120 14834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:02:12.128209 14834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:12.128283 14834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:02:12.128330 14834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:12.162827 14834 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33399
I20260620 03:02:12.162954 14981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33399 every 8 connection(s)
I20260620 03:02:12.164333 14834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:02:12.173691 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 14834
I20260620 03:02:12.174302 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42605
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:36449
--webserver_port=34377
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:02:12 dist-test-slave-f1j0 krb5kdc[14214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:02:12.189661 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:12.167499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:48005)
0620 03:02:12.168389 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:12.168439 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:12.170247 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:12.171552 (+ 1305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:12.171559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:12.172138 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:12.172668 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:12.172685 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:12.174563 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:12.174567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:12.175905 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:12.175918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:12.176988 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:12.176995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:12.177383 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:12.178146 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:12.178170 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:12.180345 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:12.185069 (+ 4724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:12.185077 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:12.185119 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:12.185559 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:12.186188 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:12.186192 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:12.186194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:12.186403 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:12.187436 (+ 1033us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:12.187442 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:12.187801 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 03:02:12.188124 (+ 323us) client_negotiation.cc:240] Negotiation successful
0620 03:02:12.188561 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":371,"threads_started":1}
I20260620 03:02:12.190205 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:12.167278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48005 (local address 127.0.97.62:36449)
0620 03:02:12.167842 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:12.167853 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:12.168590 (+ 737us) server_negotiation.cc:407] Connection header received
0620 03:02:12.170763 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:12.170776 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:12.171101 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:12.171423 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:12.172881 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:12.174308 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:12.176052 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:12.176741 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:12.176949 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:12.180669 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:12.180678 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:12.180694 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:12.180758 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:12.184886 (+ 4128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:12.185781 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:12.185793 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:12.185805 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:12.186020 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:12.186570 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:12.186578 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:12.186585 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:12.187294 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:12.187486 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:12.188408 (+ 922us) server_negotiation.cc:299] Negotiation successful
0620 03:02:12.188876 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":245,"threads_started":1}
I20260620 03:02:12.192992 14982 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36449
I20260620 03:02:12.193334 14982 heartbeater.cc:464] Registering TS with master...
I20260620 03:02:12.194352 14982 heartbeater.cc:511] Master 127.0.97.62:36449 requested a full tablet report, sending...
I20260620 03:02:12.196687 14797 ts_manager.cc:194] Registered new tserver with Master: 51a4c984db1b43ddb178038c06065720 (127.0.97.1:33399)
I20260620 03:02:12.198262 14797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48005
W20260620 03:02:12.505795 14986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:12.506167 14986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:12.506336 14986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:12.513718 14986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:12.513871 14986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:12.514065 14986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:02:12.523980 14986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34377
--tserver_master_addrs=127.0.97.62:36449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:12.525364 14986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:12.526935 14986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:12.540652 14991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:12.540699 14994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:12.540732 14992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:12.541501 14986 server_base.cc:1034] running on GCE node
I20260620 03:02:12.542071 14986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:12.542939 14986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:12.544128 14986 hybrid_clock.cc:648] HybridClock initialized: now 1781924532544076 us; error 42 us; skew 500 ppm
I20260620 03:02:12.544435 14986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:12 dist-test-slave-f1j0 krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:12.547461 14986 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:02:12.548825 14986 webserver.cc:460] Webserver started at http://127.0.97.2:34377/ using document root <none> and password file <none>
I20260620 03:02:12.549492 14986 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:12.549672 14986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:12.554981 14986 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 03:02:12.557694 15001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:12.559110 14986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 03:02:12.559278 14986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "eb7608719a3c490891988f688bd61042"
format_stamp: "Formatted at 2026-06-20 03:02:09 on dist-test-slave-f1j0"
server_key: "de6a0f6520974a81ffddbf414192f3bc"
server_key_iv: "4d4e3829ef5cda654befc53562c82739"
server_key_version: "encryptionkey@0"
I20260620 03:02:12.560321 14986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:12.628288 14986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:12.632951 14986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:12.633242 14986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:12.634353 14986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:02:12.635804 14986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:02:12.635921 14986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:12.636089 14986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:02:12.636190 14986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:12.670253 14986 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42605
I20260620 03:02:12.670276 15113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42605 every 8 connection(s)
I20260620 03:02:12.671700 14986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:02:12.682152 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 14986
I20260620 03:02:12.682862 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35411
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:36449
--webserver_port=44829
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:02:12 dist-test-slave-f1j0 krb5kdc[14214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:02:12.693632 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:12.674598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38701 (local address 127.0.97.62:36449)
0620 03:02:12.674793 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:12.674798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:12.676049 (+ 1251us) server_negotiation.cc:407] Connection header received
0620 03:02:12.677906 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:12.677912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:12.678034 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:12.678166 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:12.679657 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:12.680385 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:12.682166 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:12.682858 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:12.683008 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:12.686860 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:12.686869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:12.686877 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:12.686923 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:12.690506 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:12.691296 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:12.691302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:12.691306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:12.691440 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:12.691881 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:12.691890 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:12.691892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:12.692161 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:12.692319 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:12.693032 (+ 713us) server_negotiation.cc:299] Negotiation successful
0620 03:02:12.693267 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 03:02:12.694625 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:12.675111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:38701)
0620 03:02:12.675876 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:12.675913 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:12.677643 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:12.678402 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:12.678427 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:12.678930 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:12.679476 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:12.679495 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:12.680562 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:12.680566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:12.681980 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:12.681992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:12.683152 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:12.683159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:12.683403 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:12.684128 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:12.684156 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:12.686615 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:12.690680 (+ 4065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:12.690690 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:12.690709 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:12.691157 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:12.691560 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:12.691565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:12.691569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:12.691757 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:12.692290 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:12.692299 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:12.692661 (+ 362us) client_negotiation.cc:769] Sending connection context
0620 03:02:12.692936 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 03:02:12.693479 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":314,"threads_started":1}
I20260620 03:02:12.696199 15114 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36449
I20260620 03:02:12.696543 15114 heartbeater.cc:464] Registering TS with master...
I20260620 03:02:12.697444 15114 heartbeater.cc:511] Master 127.0.97.62:36449 requested a full tablet report, sending...
I20260620 03:02:12.699164 14797 ts_manager.cc:194] Registered new tserver with Master: eb7608719a3c490891988f688bd61042 (127.0.97.2:42605)
I20260620 03:02:12.700465 14797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38701
W20260620 03:02:13.014196 15117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:13.014467 15117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:13.014559 15117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:13.021492 15117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:13.021613 15117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:13.021724 15117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:02:13.031332 15117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44829
--tserver_master_addrs=127.0.97.62:36449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:13.032615 15117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:13.033897 15117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:13.047061 15123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:13.047464 15122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:13.047518 15125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:13.048260 15117 server_base.cc:1034] running on GCE node
I20260620 03:02:13.048816 15117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:13.049747 15117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:13.050951 15117 hybrid_clock.cc:648] HybridClock initialized: now 1781924533050915 us; error 29 us; skew 500 ppm
I20260620 03:02:13.051271 15117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:13 dist-test-slave-f1j0 krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:13.054466 15117 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:02:13.055912 15117 webserver.cc:460] Webserver started at http://127.0.97.3:44829/ using document root <none> and password file <none>
I20260620 03:02:13.056602 15117 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:13.056747 15117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:13.062105 15117 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 03:02:13.064882 15132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:13.066262 15117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:02:13.066438 15117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "72aa7ecd9ca24eacb603857634cb630e"
format_stamp: "Formatted at 2026-06-20 03:02:10 on dist-test-slave-f1j0"
server_key: "c104300cb7528619f36c7e062f9b08e6"
server_key_iv: "da772be2274f2c3b3572c1cca3722177"
server_key_version: "encryptionkey@0"
I20260620 03:02:13.067538 15117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:13.087230 15117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:13.091814 15117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:13.092100 15117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:13.093112 15117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:02:13.094635 15117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:02:13.094722 15117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:13.094892 15117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:02:13.094972 15117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:13.128918 15117 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35411
I20260620 03:02:13.129015 15244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35411 every 8 connection(s)
I20260620 03:02:13.130580 15117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:02:13.138746 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 15117
Jun 20 03:02:13 dist-test-slave-f1j0 krb5kdc[14214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:02:13.153167 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.133161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40747 (local address 127.0.97.62:36449)
0620 03:02:13.133348 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:13.133352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:13.134580 (+ 1228us) server_negotiation.cc:407] Connection header received
0620 03:02:13.136557 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:13.136563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:13.136656 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:13.136758 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:13.138281 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.139084 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.141325 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.141915 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.142030 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:13.146199 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:13.146204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:13.146208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:13.146259 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:13.149356 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:13.150454 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:13.150459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:13.150461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:13.150639 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:13.151212 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:13.151217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:13.151221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:13.151585 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:13.151732 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:13.152629 (+ 897us) server_negotiation.cc:299] Negotiation successful
0620 03:02:13.152844 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 03:02:13.154373 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.133702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:40747)
0620 03:02:13.134413 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:13.134456 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:13.136244 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:13.136976 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:13.136984 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:13.137508 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:13.138101 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:13.138119 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.139271 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.139278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:13.141158 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:13.141172 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.142172 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.142183 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:13.142493 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:13.143366 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:13.143393 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:13.145958 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:13.149602 (+ 3644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:13.149615 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:13.149637 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:13.150293 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:13.150780 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:13.150786 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:13.150792 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:13.151041 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:13.151726 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:13.151735 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:13.152205 (+ 470us) client_negotiation.cc:769] Sending connection context
0620 03:02:13.152476 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 03:02:13.152869 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":273,"threads_started":1}
I20260620 03:02:13.155356 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.139669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60382 (local address 127.0.97.62:36449)
0620 03:02:13.140207 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:13.140212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:13.140739 (+ 527us) server_negotiation.cc:407] Connection header received
0620 03:02:13.140911 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:13.140915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:13.140981 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:13.141086 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:13.142932 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.143886 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.145516 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.146016 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.146144 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:13.148792 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:13.148796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:13.148800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:13.148831 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:13.152128 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:13.152993 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:13.152998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:13.153001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:13.153147 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:13.153747 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:13.153752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:13.153755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:13.154080 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:13.154329 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:13.154529 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 03:02:13.154767 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":260,"threads_started":1}
I20260620 03:02:13.155894 15245 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36449
I20260620 03:02:13.156208 15245 heartbeater.cc:464] Registering TS with master...
I20260620 03:02:13.157130 15245 heartbeater.cc:511] Master 127.0.97.62:36449 requested a full tablet report, sending...
I20260620 03:02:13.158627 14797 ts_manager.cc:194] Registered new tserver with Master: 72aa7ecd9ca24eacb603857634cb630e (127.0.97.3:35411)
I20260620 03:02:13.159447 14797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40747
I20260620 03:02:13.170177 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:02:13.177074 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.171211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60388 (local address 127.0.97.62:36449)
0620 03:02:13.171397 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:13.171402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:13.171823 (+ 421us) server_negotiation.cc:407] Connection header received
0620 03:02:13.172060 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:13.172064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:13.172189 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:13.172373 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:02:13.172784 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.173475 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.175145 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.175557 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.175652 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:13.176114 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:02:13.176622 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:02:13.176730 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:13.176829 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 03:02:13.176893 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 03:02:13.178104 14797 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:02:13.185688 14797 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:02:13.202088 14982 heartbeater.cc:503] Master 127.0.97.62:36449 was elected leader, sending a full tablet report...
I20260620 03:02:13.211382 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.197389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34924 (local address 127.0.97.3:35411)
0620 03:02:13.198198 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:13.198208 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:13.198983 (+ 775us) server_negotiation.cc:407] Connection header received
0620 03:02:13.199957 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:13.199970 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:13.200276 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:13.200470 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:13.201533 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.204371 (+ 2838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.207861 (+ 3490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.209490 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.209647 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:13.209955 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:13.210833 (+ 878us) server_negotiation.cc:299] Negotiation successful
0620 03:02:13.211060 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":456,"threads_started":1}
I20260620 03:02:13.212282 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.197049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35411 (local address 127.0.0.1:34924)
0620 03:02:13.198806 (+ 1757us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:13.198864 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:13.199747 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:13.200841 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:13.200851 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:13.200882 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:13.201320 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:13.201335 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.204591 (+ 3256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.204596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:13.207622 (+ 3026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:13.207645 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.210187 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.210196 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:13.210481 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:13.210552 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 03:02:13.211687 (+ 1135us) client_negotiation.cc:240] Negotiation successful
0620 03:02:13.211886 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1459,"thread_start_us":1343,"threads_started":1}
I20260620 03:02:13.216444 15180 tablet_service.cc:1467] Processing CreateTablet for tablet 24d35379df60451998a4999c10ac45be (DEFAULT_TABLE table=test-table [id=97b4445081b74bd884504212bb5998d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:02:13.218055 15180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24d35379df60451998a4999c10ac45be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:02:13.218945 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.197686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42605 (local address 127.0.0.1:46010)
0620 03:02:13.199362 (+ 1676us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:13.199384 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:13.199542 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:13.212187 (+ 12645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:13.212193 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:13.212235 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:13.212632 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:13.212645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.214574 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.214582 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:13.216286 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:13.216300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.218119 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.218125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:13.218339 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:13.218379 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 03:02:13.218478 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 03:02:13.218614 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1562,"thread_start_us":1428,"threads_started":1}
I20260620 03:02:13.219581 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.197959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46010 (local address 127.0.97.2:42605)
0620 03:02:13.211550 (+ 13591us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:13.211558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:13.211587 (+ 29us) server_negotiation.cc:407] Connection header received
0620 03:02:13.211685 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:13.211692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:13.211958 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:13.212836 (+ 878us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:13.213045 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.214435 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.216465 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.217905 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.218037 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:13.218232 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:13.218549 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 03:02:13.218714 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13447,"thread_start_us":13327,"threads_started":1}
I20260620 03:02:13.220124 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.199419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:43212)
0620 03:02:13.209654 (+ 10235us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:13.209684 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:13.209824 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:13.212838 (+ 3014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:13.212842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:13.212867 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:13.213498 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:13.213509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.215465 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.215471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:13.217098 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:13.217109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.219370 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.219379 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:13.219543 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:13.219575 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 03:02:13.219666 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 03:02:13.219787 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10060,"thread_start_us":10495,"threads_started":1}
I20260620 03:02:13.220800 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.199738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43212 (local address 127.0.97.1:33399)
0620 03:02:13.212280 (+ 12542us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:13.212286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:13.212308 (+ 22us) server_negotiation.cc:407] Connection header received
0620 03:02:13.212425 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:13.212435 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:13.212642 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:13.212859 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:13.213742 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.215319 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.217317 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.218641 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.218774 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:13.218987 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:13.220456 (+ 1469us) server_negotiation.cc:299] Negotiation successful
0620 03:02:13.220573 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12396,"thread_start_us":12257,"threads_started":1}
I20260620 03:02:13.223485 14917 tablet_service.cc:1467] Processing CreateTablet for tablet 24d35379df60451998a4999c10ac45be (DEFAULT_TABLE table=test-table [id=97b4445081b74bd884504212bb5998d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:02:13.223548 15049 tablet_service.cc:1467] Processing CreateTablet for tablet 24d35379df60451998a4999c10ac45be (DEFAULT_TABLE table=test-table [id=97b4445081b74bd884504212bb5998d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:02:13.224680 14917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24d35379df60451998a4999c10ac45be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:02:13.224822 15049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24d35379df60451998a4999c10ac45be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:02:13.233847 15261 tablet_bootstrap.cc:492] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e: Bootstrap starting.
I20260620 03:02:13.237946 15261 tablet_bootstrap.cc:654] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e: Neither blocks nor log segments found. Creating new log.
I20260620 03:02:13.239351 15261 log.cc:824] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:13.239593 15262 tablet_bootstrap.cc:492] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042: Bootstrap starting.
I20260620 03:02:13.242265 15263 tablet_bootstrap.cc:492] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720: Bootstrap starting.
I20260620 03:02:13.243472 15262 tablet_bootstrap.cc:654] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042: Neither blocks nor log segments found. Creating new log.
I20260620 03:02:13.244714 15262 log.cc:824] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:13.245823 15263 tablet_bootstrap.cc:654] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720: Neither blocks nor log segments found. Creating new log.
I20260620 03:02:13.247008 15263 log.cc:824] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:13.248557 15262 tablet_bootstrap.cc:492] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042: No bootstrap required, opened a new log
I20260620 03:02:13.248835 15262 ts_tablet_manager.cc:1397] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 03:02:13.249881 15261 tablet_bootstrap.cc:492] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e: No bootstrap required, opened a new log
I20260620 03:02:13.249927 15263 tablet_bootstrap.cc:492] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720: No bootstrap required, opened a new log
I20260620 03:02:13.250159 15263 ts_tablet_manager.cc:1397] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 03:02:13.250161 15261 ts_tablet_manager.cc:1397] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
I20260620 03:02:13.255880 15262 raft_consensus.cc:348] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } }
I20260620 03:02:13.256310 15262 raft_consensus.cc:374] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:02:13.256417 15262 raft_consensus.cc:729] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7608719a3c490891988f688bd61042, State: Initialized, Role: FOLLOWER
I20260620 03:02:13.257063 15262 consensus_queue.cc:260] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } }
I20260620 03:02:13.258193 15114 heartbeater.cc:503] Master 127.0.97.62:36449 was elected leader, sending a full tablet report...
I20260620 03:02:13.260965 15263 raft_consensus.cc:348] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } }
I20260620 03:02:13.260964 15261 raft_consensus.cc:348] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } }
I20260620 03:02:13.261404 15263 raft_consensus.cc:374] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:02:13.261402 15261 raft_consensus.cc:374] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:02:13.261508 15263 raft_consensus.cc:729] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a4c984db1b43ddb178038c06065720, State: Initialized, Role: FOLLOWER
I20260620 03:02:13.261512 15261 raft_consensus.cc:729] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72aa7ecd9ca24eacb603857634cb630e, State: Initialized, Role: FOLLOWER
I20260620 03:02:13.262151 15263 consensus_queue.cc:260] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } }
I20260620 03:02:13.262151 15261 consensus_queue.cc:260] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } }
I20260620 03:02:13.262504 15262 ts_tablet_manager.cc:1428] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
I20260620 03:02:13.263648 15245 heartbeater.cc:503] Master 127.0.97.62:36449 was elected leader, sending a full tablet report...
I20260620 03:02:13.264504 15261 ts_tablet_manager.cc:1428] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260620 03:02:13.266075 15263 ts_tablet_manager.cc:1428] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260620 03:02:13.356985 15269 raft_consensus.cc:482] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:02:13.357309 15269 raft_consensus.cc:504] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } }
I20260620 03:02:13.359102 15269 leader_election.cc:288] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51a4c984db1b43ddb178038c06065720 (127.0.97.1:33399), eb7608719a3c490891988f688bd61042 (127.0.97.2:42605)
I20260620 03:02:13.365273 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.359556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56217 (local address 127.0.97.1:33399)
0620 03:02:13.359747 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:13.359754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:13.359845 (+ 91us) server_negotiation.cc:407] Connection header received
0620 03:02:13.360069 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:13.360075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:13.360171 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:13.360275 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:13.360811 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.361529 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.363559 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.364399 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.364499 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:13.364630 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:13.364964 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 03:02:13.365072 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 03:02:13.365346 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.359534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.97.3:56217)
0620 03:02:13.359743 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:13.359762 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:13.359928 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:13.360308 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:13.360312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:13.360339 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:13.360638 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:13.360659 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.361710 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.361714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:13.363396 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:13.363410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.364544 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.364551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:13.364748 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:13.364807 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 03:02:13.364919 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 03:02:13.365057 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":15}
I20260620 03:02:13.366467 14937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d35379df60451998a4999c10ac45be" candidate_uuid: "72aa7ecd9ca24eacb603857634cb630e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a4c984db1b43ddb178038c06065720" is_pre_election: true
I20260620 03:02:13.366799 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.359550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42605 (local address 127.0.97.3:60047)
0620 03:02:13.360846 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:13.360860 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:13.360988 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:13.361310 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:13.361315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:13.361339 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:13.361693 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:13.361704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.362777 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.362782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:13.364517 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:13.364528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.366149 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.366161 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:13.366282 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:13.366304 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 03:02:13.366396 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 03:02:13.366482 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"mutex_wait_us":508,"thread_start_us":467,"threads_started":1}
I20260620 03:02:13.366838 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.359622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60047 (local address 127.0.97.2:42605)
0620 03:02:13.359846 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:13.359851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:13.360947 (+ 1096us) server_negotiation.cc:407] Connection header received
0620 03:02:13.361130 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:13.361134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:13.361191 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:13.361289 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:13.361886 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.362613 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.364685 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.365878 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.366049 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:13.366147 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:13.366459 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 03:02:13.366596 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 03:02:13.367089 14937 raft_consensus.cc:2454] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72aa7ecd9ca24eacb603857634cb630e in term 0.
I20260620 03:02:13.367930 15135 leader_election.cc:302] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a4c984db1b43ddb178038c06065720, 72aa7ecd9ca24eacb603857634cb630e; no voters:
I20260620 03:02:13.368046 15069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d35379df60451998a4999c10ac45be" candidate_uuid: "72aa7ecd9ca24eacb603857634cb630e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb7608719a3c490891988f688bd61042" is_pre_election: true
I20260620 03:02:13.368372 15269 raft_consensus.cc:2781] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:02:13.368546 15269 raft_consensus.cc:482] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:02:13.368537 15069 raft_consensus.cc:2454] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72aa7ecd9ca24eacb603857634cb630e in term 0.
I20260620 03:02:13.368670 15269 raft_consensus.cc:3037] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:02:13.371017 15269 raft_consensus.cc:504] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } }
I20260620 03:02:13.371605 15269 leader_election.cc:288] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [CANDIDATE]: Term 1 election: Requested vote from peers 51a4c984db1b43ddb178038c06065720 (127.0.97.1:33399), eb7608719a3c490891988f688bd61042 (127.0.97.2:42605)
I20260620 03:02:13.372074 15069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d35379df60451998a4999c10ac45be" candidate_uuid: "72aa7ecd9ca24eacb603857634cb630e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb7608719a3c490891988f688bd61042"
I20260620 03:02:13.372124 14937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d35379df60451998a4999c10ac45be" candidate_uuid: "72aa7ecd9ca24eacb603857634cb630e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a4c984db1b43ddb178038c06065720"
I20260620 03:02:13.372290 15069 raft_consensus.cc:3037] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:02:13.372323 14937 raft_consensus.cc:3037] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:02:13.374620 15069 raft_consensus.cc:2454] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72aa7ecd9ca24eacb603857634cb630e in term 1.
I20260620 03:02:13.374624 14937 raft_consensus.cc:2454] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72aa7ecd9ca24eacb603857634cb630e in term 1.
I20260620 03:02:13.375068 15135 leader_election.cc:302] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a4c984db1b43ddb178038c06065720, 72aa7ecd9ca24eacb603857634cb630e; no voters:
I20260620 03:02:13.375389 15269 raft_consensus.cc:2781] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:02:13.375835 15269 raft_consensus.cc:686] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [term 1 LEADER]: Becoming Leader. State: Replica: 72aa7ecd9ca24eacb603857634cb630e, State: Running, Role: LEADER
I20260620 03:02:13.376402 15269 consensus_queue.cc:237] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } }
I20260620 03:02:13.380812 14796 catalog_manager.cc:5495] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e reported cstate change: term changed from 0 to 1, leader changed from <none> to 72aa7ecd9ca24eacb603857634cb630e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "72aa7ecd9ca24eacb603857634cb630e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } health_report { overall_health: UNKNOWN } } }
W20260620 03:02:13.382148 15246 tablet.cc:2369] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:02:13.399040 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.393547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34928 (local address 127.0.97.3:35411)
0620 03:02:13.393744 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:13.393752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:13.393776 (+ 24us) server_negotiation.cc:407] Connection header received
0620 03:02:13.394021 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:13.394025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:13.394102 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:13.394261 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:02:13.394615 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.395348 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.396838 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.397239 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.397370 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:13.397979 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:02:13.398476 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:02:13.398649 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:13.398700 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 03:02:13.398781 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 03:02:13.407182 14937 raft_consensus.cc:1261] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Refusing update from remote peer 72aa7ecd9ca24eacb603857634cb630e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:02:13.407186 15069 raft_consensus.cc:1261] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042 [term 1 FOLLOWER]: Refusing update from remote peer 72aa7ecd9ca24eacb603857634cb630e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:02:13.408180 15271 consensus_queue.cc:1035] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:02:13.408533 15269 consensus_queue.cc:1035] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 03:02:13.416641 14983 tablet.cc:2369] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:02:13.419900 15275 mvcc.cc:205] Tried to move back new op lower bound from 7298762888825495552 to 7298762888713736192. Current Snapshot: MvccSnapshot[applied={T|T < 7298762888825495552}]
W20260620 03:02:13.424041 15115 tablet.cc:2369] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:02:13.430418 14796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:02:13.430724 14796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:02:13.434571 14796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598: Sending DeleteTablet for 3 replicas of tablet 24d35379df60451998a4999c10ac45be
I20260620 03:02:13.435969 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 14834
I20260620 03:02:13.435917 15180 tablet_service.cc:1514] Processing DeleteTablet for tablet 24d35379df60451998a4999c10ac45be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:02:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34924
I20260620 03:02:13.435896 15049 tablet_service.cc:1514] Processing DeleteTablet for tablet 24d35379df60451998a4999c10ac45be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:02:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46010
I20260620 03:02:13.436655 14917 tablet_service.cc:1514] Processing DeleteTablet for tablet 24d35379df60451998a4999c10ac45be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:02:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43212
I20260620 03:02:13.450729 15286 tablet_replica.cc:330] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720: stopping tablet replica
I20260620 03:02:13.451241 15286 raft_consensus.cc:2229] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:02:13.451820 15286 raft_consensus.cc:2258] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:02:13.453490 15284 tablet_replica.cc:330] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e: stopping tablet replica
I20260620 03:02:13.454145 15284 raft_consensus.cc:2229] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:02:13.454928 15284 raft_consensus.cc:2258] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:02:13.455739 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.436804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.0.97.1:33399)
0620 03:02:13.436977 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:13.436986 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:13.437029 (+ 43us) server_negotiation.cc:407] Connection header received
0620 03:02:13.437192 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:13.437197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:13.437271 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:13.437393 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:13.439128 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.440498 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.442943 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.443787 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.443930 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:13.447355 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:13.447380 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:13.447404 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:13.447456 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:13.452507 (+ 5051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:13.453418 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:13.453427 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:13.453430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:13.453621 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:13.454036 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:13.454041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:13.454044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:13.454499 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:13.454792 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:13.455091 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 03:02:13.455324 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 03:02:13.456391 14977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:13.457324 15284 ts_tablet_manager.cc:1905] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:02:13.460554 15285 tablet_replica.cc:330] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042: stopping tablet replica
I20260620 03:02:13.462152 15285 raft_consensus.cc:2229] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:02:13.462636 15285 raft_consensus.cc:2258] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:02:13.464856 15285 ts_tablet_manager.cc:1905] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:02:13.467233 15284 ts_tablet_manager.cc:1918] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:02:13.467422 15284 log.cc:1196] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/24d35379df60451998a4999c10ac45be
I20260620 03:02:13.467993 15284 ts_tablet_manager.cc:1939] T 24d35379df60451998a4999c10ac45be P 72aa7ecd9ca24eacb603857634cb630e: Deleting consensus metadata
I20260620 03:02:13.470037 14783 catalog_manager.cc:4841] TS 72aa7ecd9ca24eacb603857634cb630e (127.0.97.3:35411): tablet 24d35379df60451998a4999c10ac45be (table test-table [id=97b4445081b74bd884504212bb5998d3]) successfully deleted
I20260620 03:02:13.472229 15285 ts_tablet_manager.cc:1918] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:02:13.472393 15285 log.cc:1196] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/24d35379df60451998a4999c10ac45be
I20260620 03:02:13.472944 15285 ts_tablet_manager.cc:1939] T 24d35379df60451998a4999c10ac45be P eb7608719a3c490891988f688bd61042: Deleting consensus metadata
I20260620 03:02:13.474753 14782 catalog_manager.cc:4841] TS eb7608719a3c490891988f688bd61042 (127.0.97.2:42605): tablet 24d35379df60451998a4999c10ac45be (table test-table [id=97b4445081b74bd884504212bb5998d3]) successfully deleted
I20260620 03:02:13.550788 15286 ts_tablet_manager.cc:1905] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:02:13.551632 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 14834
W20260620 03:02:13.576074 14783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33399 (error 108)
I20260620 03:02:13.576378 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 14986
I20260620 03:02:13.578692 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.578296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:57146)
0620 03:02:13.578462 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:13.578560 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 03:02:13.579001 14783 catalog_manager.cc:4584] TS 51a4c984db1b43ddb178038c06065720 (127.0.97.1:33399): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 24d35379df60451998a4999c10ac45be: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33399: connect: Connection refused (error 111)
I20260620 03:02:13.597754 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.577233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47840 (local address 127.0.97.2:42605)
0620 03:02:13.577423 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:13.577429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:13.577450 (+ 21us) server_negotiation.cc:407] Connection header received
0620 03:02:13.577702 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:13.577707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:13.577819 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:13.577932 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:13.579584 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.580330 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.582347 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.582720 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.582827 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:13.591272 (+ 8445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:13.591289 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:13.591305 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:13.591342 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:13.594710 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:13.595492 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:13.595500 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:13.595502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:13.595625 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:13.596024 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:13.596027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:13.596029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:13.596380 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:13.596582 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:13.597188 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 03:02:13.597347 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 03:02:13.598496 15109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:13.632890 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.632318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:57148)
0620 03:02:13.632574 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:13.632696 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 03:02:13.634116 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.633698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:57150)
0620 03:02:13.633861 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:13.633952 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 03:02:13.691220 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 14986
I20260620 03:02:13.694707 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.694146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:57160)
0620 03:02:13.694414 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:13.694525 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 03:02:13.695932 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.695493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:57168)
0620 03:02:13.695686 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:13.695782 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 03:02:13.709728 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 15117
I20260620 03:02:13.726773 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.710629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36664 (local address 127.0.97.3:35411)
0620 03:02:13.710816 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:13.710820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:13.710928 (+ 108us) server_negotiation.cc:407] Connection header received
0620 03:02:13.711120 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:13.711124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:13.711203 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:13.711290 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:13.713150 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.714293 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.716227 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:13.716670 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:13.716756 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:13.720006 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:13.720017 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:13.720031 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:13.720077 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:13.723038 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:13.723928 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:13.723942 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:13.723945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:13.724222 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:13.724760 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:13.724783 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:13.724787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:13.725277 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:13.725647 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:13.726028 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 03:02:13.726336 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 03:02:13.728073 15240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:13.776721 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.776147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:57176)
0620 03:02:13.776401 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:13.776528 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 03:02:13.777880 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.777401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:57188)
0620 03:02:13.777656 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:13.777737 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 03:02:13.848743 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 15117
I20260620 03:02:13.867920 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 14764
I20260620 03:02:13.869134 14827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:13.982187 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:13.981541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:57202)
0620 03:02:13.981744 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:13.981847 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 03:02:13.982249 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 14764
I20260620 03:02:14.004129 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36449
--webserver_interface=127.0.97.62
--webserver_port=34355
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:02:14.338848 15292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:14.339140 15292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:14.339319 15292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:14.346390 15292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:02:14.346525 15292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:14.346612 15292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:14.346716 15292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:02:14.346792 15292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:02:14.356839 15292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:14.358305 15292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:14.359601 15292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:14.371757 15297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:14.371714 15298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:14.371824 15300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:14.372541 15292 server_base.cc:1034] running on GCE node
I20260620 03:02:14.373311 15292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:14.374792 15292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:14.375990 15292 hybrid_clock.cc:648] HybridClock initialized: now 1781924534375957 us; error 31 us; skew 500 ppm
I20260620 03:02:14.376312 15292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:14 dist-test-slave-f1j0 krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:14.379513 15292 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:02:14.381038 15292 webserver.cc:460] Webserver started at http://127.0.97.62:34355/ using document root <none> and password file <none>
I20260620 03:02:14.381800 15292 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:14.381934 15292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:14.387454 15292 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 03:02:14.390586 15307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:14.391866 15292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 03:02:14.392052 15292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "c9a6bd4ede4241cf9c7581517022d598"
format_stamp: "Formatted at 2026-06-20 03:02:08 on dist-test-slave-f1j0"
server_key: "6fd6b90e8692b03cfc225221b20b865e"
server_key_iv: "d0dfd3853a4b964cf0f2e0f3b3ba1f87"
server_key_version: "encryptionkey@0"
I20260620 03:02:14.393049 15292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:14.406631 15292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:14.410627 15292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:14.410923 15292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:14.429523 15292 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36449
I20260620 03:02:14.429543 15358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36449 every 8 connection(s)
I20260620 03:02:14.431056 15292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:02:14.437776 15359 sys_catalog.cc:263] Verifying existing consensus state
I20260620 03:02:14.439064 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 15292
I20260620 03:02:14.439883 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33399
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:36449
--webserver_port=45481
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:02:14.441071 15359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598: Bootstrap starting.
I20260620 03:02:14.460868 15359 log.cc:824] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:14.476454 15359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 03:02:14.477051 15359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598: Bootstrap complete.
I20260620 03:02:14.486371 15359 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a6bd4ede4241cf9c7581517022d598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36449 } }
I20260620 03:02:14.487390 15359 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a6bd4ede4241cf9c7581517022d598, State: Initialized, Role: FOLLOWER
I20260620 03:02:14.488109 15359 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a6bd4ede4241cf9c7581517022d598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36449 } }
I20260620 03:02:14.488371 15359 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:02:14.488543 15359 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:02:14.488732 15359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 2 FOLLOWER]: Advancing to term 3
I20260620 03:02:14.492664 15359 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a6bd4ede4241cf9c7581517022d598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36449 } }
I20260620 03:02:14.493182 15359 leader_election.cc:302] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9a6bd4ede4241cf9c7581517022d598; no voters:
I20260620 03:02:14.493906 15359 leader_election.cc:288] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 03:02:14.494047 15363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 03:02:14.495234 15363 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [term 3 LEADER]: Becoming Leader. State: Replica: c9a6bd4ede4241cf9c7581517022d598, State: Running, Role: LEADER
I20260620 03:02:14.495796 15363 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a6bd4ede4241cf9c7581517022d598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36449 } }
I20260620 03:02:14.496161 15359 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:02:14.502241 15364 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c9a6bd4ede4241cf9c7581517022d598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a6bd4ede4241cf9c7581517022d598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36449 } } }
I20260620 03:02:14.502766 15364 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [sys.catalog]: This master's current role is: LEADER
I20260620 03:02:14.502813 15365 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9a6bd4ede4241cf9c7581517022d598. Latest consensus state: current_term: 3 leader_uuid: "c9a6bd4ede4241cf9c7581517022d598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a6bd4ede4241cf9c7581517022d598" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36449 } } }
I20260620 03:02:14.503082 15365 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9a6bd4ede4241cf9c7581517022d598 [sys.catalog]: This master's current role is: LEADER
I20260620 03:02:14.513072 15372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:02:14.521505 15372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:02:14.522970 15372 catalog_manager.cc:1259] Loaded cluster ID: 7f4e1a1298ad40609d0e0e19407c5a20
I20260620 03:02:14.523073 15372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 03:02:14.525053 15372 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f550a3f99ad google::LogMessage::Fail() at ??:0
@ 0x7f550a3fc88a google::LogMessage::SendToLog() at ??:0
@ 0x7f550a3f94de google::LogMessage::Flush() at ??:0
@ 0x7f550a3faa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f551fe2c512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f551fe29976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f551feb1610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f551feb1439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f550d045345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f550d04891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f550d048799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f550d023eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f55102fd6db start_thread at ??:0
@ 0x7f550929361f clone at ??:0
W20260620 03:02:14.812095 15361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:14.812440 15361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:14.812618 15361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:14.820026 15361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:14.820237 15361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:14.820418 15361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:02:14.830440 15361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45481
--tserver_master_addrs=127.0.97.62:36449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:14.831781 15361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:14.833287 15361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:14.846208 15400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:14.846256 15401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:14.846330 15403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:14.847455 15361 server_base.cc:1034] running on GCE node
I20260620 03:02:14.848062 15361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:14.849027 15361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:14.850282 15361 hybrid_clock.cc:648] HybridClock initialized: now 1781924534850230 us; error 63 us; skew 500 ppm
I20260620 03:02:14.850611 15361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:14 dist-test-slave-f1j0 krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:14.853710 15361 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:02:14.855219 15361 webserver.cc:460] Webserver started at http://127.0.97.1:45481/ using document root <none> and password file <none>
I20260620 03:02:14.855901 15361 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:14.856046 15361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:14.861326 15361 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:02:14.864219 15410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:14.865811 15361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 03:02:14.865985 15361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "51a4c984db1b43ddb178038c06065720"
format_stamp: "Formatted at 2026-06-20 03:02:09 on dist-test-slave-f1j0"
server_key: "398ec8d9d132bd3c2338480a6f86b874"
server_key_iv: "269704f795ae101a75b46a28812d3f69"
server_key_version: "encryptionkey@0"
I20260620 03:02:14.866958 15361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:14.881454 15361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:14.885747 15361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:14.886022 15361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:14.886986 15361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:02:14.888828 15417 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 03:02:14.894443 15361 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 03:02:14.894591 15361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 03:02:14.894747 15361 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 03:02:14.897969 15361 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 03:02:14.898075 15361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.003s
I20260620 03:02:14.898411 15417 tablet_bootstrap.cc:492] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720: Bootstrap starting.
I20260620 03:02:14.908660 15417 log.cc:824] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:14.921610 15417 tablet_bootstrap.cc:492] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 03:02:14.922611 15417 tablet_bootstrap.cc:492] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720: Bootstrap complete.
I20260620 03:02:14.923100 15417 ts_tablet_manager.cc:1397] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.007s
I20260620 03:02:14.930644 15417 raft_consensus.cc:348] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } }
I20260620 03:02:14.932195 15417 raft_consensus.cc:729] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a4c984db1b43ddb178038c06065720, State: Initialized, Role: FOLLOWER
I20260620 03:02:14.932973 15417 consensus_queue.cc:260] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } }
I20260620 03:02:14.934609 15417 ts_tablet_manager.cc:1428] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260620 03:02:14.950096 15361 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33399
I20260620 03:02:14.950142 15525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33399 every 8 connection(s)
I20260620 03:02:14.951373 15361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:02:14.953150 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 15361
I20260620 03:02:14.953866 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42605
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:36449
--webserver_port=34377
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:02:14.956445 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:14.954384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:40541)
0620 03:02:14.955284 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:14.955516 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":407,"threads_started":1}
W20260620 03:02:14.957077 15526 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36449 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:36449: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
I20260620 03:02:14.957994 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:14.957675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:54411)
0620 03:02:14.957821 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:14.957888 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 03:02:14.958930 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:14.958630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:34977)
0620 03:02:14.958755 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:14.958821 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 03:02:14.959407 15526 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 03:02:15.282399 15529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:15.282682 15529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:15.282891 15529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:15.290223 15529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:15.290323 15529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:15.290529 15529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:02:15.299979 15529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34377
--tserver_master_addrs=127.0.97.62:36449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:15.301297 15529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:15.302642 15529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:15.316663 15534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:15.316689 15537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:15.316752 15535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:15.317451 15529 server_base.cc:1034] running on GCE node
I20260620 03:02:15.318099 15529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:15.319056 15529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:15.320258 15529 hybrid_clock.cc:648] HybridClock initialized: now 1781924535320204 us; error 41 us; skew 500 ppm
I20260620 03:02:15.320538 15529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:15 dist-test-slave-f1j0 krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:15.323586 15529 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:02:15.325119 15529 webserver.cc:460] Webserver started at http://127.0.97.2:34377/ using document root <none> and password file <none>
I20260620 03:02:15.325896 15529 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:15.326042 15529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:15.331314 15529 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 03:02:15.334087 15544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:15.335347 15529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:02:15.335529 15529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "eb7608719a3c490891988f688bd61042"
format_stamp: "Formatted at 2026-06-20 03:02:09 on dist-test-slave-f1j0"
server_key: "de6a0f6520974a81ffddbf414192f3bc"
server_key_iv: "4d4e3829ef5cda654befc53562c82739"
server_key_version: "encryptionkey@0"
I20260620 03:02:15.336515 15529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:15.359583 15529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:15.363830 15529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:15.364110 15529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:15.365091 15529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:02:15.366603 15529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:02:15.366731 15529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:15.366875 15529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:02:15.366966 15529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:15.400755 15529 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42605
I20260620 03:02:15.400843 15656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42605 every 8 connection(s)
I20260620 03:02:15.402333 15529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:02:15.406822 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 15529
I20260620 03:02:15.407231 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:15.405322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:47941)
0620 03:02:15.406097 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:15.406297 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":349,"threads_started":1}
I20260620 03:02:15.407568 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35411
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:36449
--webserver_port=44829
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:02:15.407971 15657 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36449 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:36449: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
I20260620 03:02:15.408813 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:15.408497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:54605)
0620 03:02:15.408638 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:15.408701 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 03:02:15.409747 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:15.409397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:56535)
0620 03:02:15.409518 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:15.409646 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 03:02:15.410033 15657 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 03:02:15.733143 15660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:15.733587 15660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:15.733803 15660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:15.741463 15660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:15.741645 15660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:15.741824 15660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:02:15.751708 15660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44829
--tserver_master_addrs=127.0.97.62:36449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:15.753067 15660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:15.754451 15660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:15.767464 15665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:15.767498 15668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:15.767570 15666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:15.768116 15660 server_base.cc:1034] running on GCE node
I20260620 03:02:15.768718 15660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:15.769723 15660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:15.770920 15660 hybrid_clock.cc:648] HybridClock initialized: now 1781924535770856 us; error 58 us; skew 500 ppm
I20260620 03:02:15.771301 15660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:15 dist-test-slave-f1j0 krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:15.774356 15660 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:02:15.775733 15660 webserver.cc:460] Webserver started at http://127.0.97.3:44829/ using document root <none> and password file <none>
I20260620 03:02:15.776470 15660 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:15.776614 15660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:15.781916 15660 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 03:02:15.784490 15675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:15.785913 15660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:02:15.786098 15660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "72aa7ecd9ca24eacb603857634cb630e"
format_stamp: "Formatted at 2026-06-20 03:02:10 on dist-test-slave-f1j0"
server_key: "c104300cb7528619f36c7e062f9b08e6"
server_key_iv: "da772be2274f2c3b3572c1cca3722177"
server_key_version: "encryptionkey@0"
I20260620 03:02:15.787132 15660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:15.810333 15660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:15.814431 15660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:15.814706 15660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:15.815732 15660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:02:15.817101 15660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:02:15.817233 15660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:15.817370 15660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:02:15.817458 15660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:15.850020 15660 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35411
I20260620 03:02:15.850106 15787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35411 every 8 connection(s)
I20260620 03:02:15.851445 15660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:02:15.856180 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:15.854417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:35091)
0620 03:02:15.855135 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:15.855351 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":281,"threads_started":1}
W20260620 03:02:15.856825 15788 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36449 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:36449: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
I20260620 03:02:15.857710 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:15.857343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:43459)
0620 03:02:15.857492 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:15.857564 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 03:02:15.858522 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:15.858200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:36225)
0620 03:02:15.858332 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:15.858418 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 03:02:15.858747 15788 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 03:02:15.860168 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 15660
I20260620 03:02:15.960731 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:15.959883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:36595)
0620 03:02:15.960370 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:15.960452 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":229,"threads_started":1}
I20260620 03:02:16.411330 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:16.410569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:59833)
0620 03:02:16.411013 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:16.411103 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":217,"threads_started":1}
I20260620 03:02:16.654949 15794 raft_consensus.cc:482] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:02:16.655225 15794 raft_consensus.cc:504] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } }
I20260620 03:02:16.656833 15794 leader_election.cc:288] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72aa7ecd9ca24eacb603857634cb630e (127.0.97.3:35411), eb7608719a3c490891988f688bd61042 (127.0.97.2:42605)
Jun 20 03:02:16 dist-test-slave-f1j0 krb5kdc[14214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Jun 20 03:02:16 dist-test-slave-f1j0 krb5kdc[14214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:02:16.690990 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:16.657334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40925 (local address 127.0.97.3:35411)
0620 03:02:16.669937 (+ 12603us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:16.669958 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:16.670003 (+ 45us) server_negotiation.cc:407] Connection header received
0620 03:02:16.671794 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:16.671804 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:16.672214 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:16.672566 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:16.674550 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:16.676002 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:16.677838 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:16.678462 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:16.678656 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:16.682936 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:16.682942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:16.682955 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:16.682993 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:16.686186 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:16.687856 (+ 1670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:16.687869 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:16.687895 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:16.688161 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:16.688525 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:16.688530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:16.688533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:16.689209 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:16.689372 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:16.690171 (+ 799us) server_negotiation.cc:299] Negotiation successful
0620 03:02:16.690549 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12432,"thread_start_us":12329,"threads_started":1}
I20260620 03:02:16.691126 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:16.657651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41503 (local address 127.0.97.2:42605)
0620 03:02:16.670762 (+ 13111us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:16.670771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:16.670808 (+ 37us) server_negotiation.cc:407] Connection header received
0620 03:02:16.672364 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:16.672373 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:16.672701 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:16.673002 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:16.674544 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:16.675913 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:16.677839 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:16.678466 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:16.678657 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:16.682764 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:16.682781 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:16.682798 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:16.682841 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:16.686021 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:16.687299 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:16.687310 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:16.687319 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:16.687615 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:16.688131 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:16.688137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:16.688142 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:16.688977 (+ 835us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:16.689208 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:16.690119 (+ 911us) server_negotiation.cc:299] Negotiation successful
0620 03:02:16.690621 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12948,"thread_start_us":12811,"threads_started":1}
I20260620 03:02:16.691037 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:16.657120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35411 (local address 127.0.97.1:40925)
0620 03:02:16.667882 (+ 10762us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:16.667939 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:16.668772 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:16.672706 (+ 3934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:16.672717 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:16.673734 (+ 1017us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:16.674349 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:16.674369 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:16.676185 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:16.676194 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:16.677698 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:16.677710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:16.678733 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:16.678740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:16.679043 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:16.679831 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:16.679858 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:16.682652 (+ 2794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:16.686332 (+ 3680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:16.686339 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:16.686345 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:16.687104 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:16.688270 (+ 1166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:16.688273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:16.688276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:16.688400 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:16.689330 (+ 930us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:16.689334 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:16.689500 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 03:02:16.689848 (+ 348us) client_negotiation.cc:240] Negotiation successful
0620 03:02:16.690514 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10554,"mutex_wait_us":197,"spinlock_wait_cycles":37120,"thread_start_us":12887,"threads_started":1}
I20260620 03:02:16.691059 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:16.657361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42605 (local address 127.0.97.1:41503)
0620 03:02:16.670320 (+ 12959us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:16.670340 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:16.670448 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:16.672965 (+ 2517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:16.672970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:16.673759 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:16.674350 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:16.674370 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:16.676137 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:16.676141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:16.677695 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:16.677708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:16.678670 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:16.678676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:16.679049 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:16.679895 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:16.679923 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:16.682447 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:16.686191 (+ 3744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:16.686202 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:16.686218 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:16.687104 (+ 886us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:16.687750 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:16.687758 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:16.687763 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:16.687977 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:16.689129 (+ 1152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:16.689135 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:16.689500 (+ 365us) client_negotiation.cc:769] Sending connection context
0620 03:02:16.689848 (+ 348us) client_negotiation.cc:240] Negotiation successful
0620 03:02:16.690646 (+ 798us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12836,"mutex_wait_us":467,"spinlock_wait_cycles":13952,"thread_start_us":11838,"threads_started":1}
W20260620 03:02:16.694428 15412 leader_election.cc:341] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb7608719a3c490891988f688bd61042 (127.0.97.2:42605): Not found: Tablet not found: 24d35379df60451998a4999c10ac45be
W20260620 03:02:16.694661 15413 leader_election.cc:341] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72aa7ecd9ca24eacb603857634cb630e (127.0.97.3:35411): Not found: Tablet not found: 24d35379df60451998a4999c10ac45be
I20260620 03:02:16.694878 15413 leader_election.cc:302] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51a4c984db1b43ddb178038c06065720; no voters: 72aa7ecd9ca24eacb603857634cb630e, eb7608719a3c490891988f688bd61042
I20260620 03:02:16.695293 15794 raft_consensus.cc:2726] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 03:02:16.860054 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:16.859233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:41465)
0620 03:02:16.859693 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:16.859822 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":220,"threads_started":1}
I20260620 03:02:16.962220 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:16.961801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:57635)
0620 03:02:16.961983 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:16.962087 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 03:02:17.412883 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:17.412216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:37583)
0620 03:02:17.412631 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:17.412708 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":222,"threads_started":1}
I20260620 03:02:17.861526 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:17.860890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:32905)
0620 03:02:17.861298 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:17.861373 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":217,"threads_started":1}
I20260620 03:02:17.963827 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:17.963066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:58385)
0620 03:02:17.963560 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:17.963637 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":275,"threads_started":1}
I20260620 03:02:18.255378 15803 raft_consensus.cc:482] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:02:18.255561 15803 raft_consensus.cc:504] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } }
I20260620 03:02:18.256124 15803 leader_election.cc:288] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72aa7ecd9ca24eacb603857634cb630e (127.0.97.3:35411), eb7608719a3c490891988f688bd61042 (127.0.97.2:42605)
W20260620 03:02:18.257151 15413 leader_election.cc:341] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72aa7ecd9ca24eacb603857634cb630e (127.0.97.3:35411): Not found: Tablet not found: 24d35379df60451998a4999c10ac45be
W20260620 03:02:18.257341 15412 leader_election.cc:341] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb7608719a3c490891988f688bd61042 (127.0.97.2:42605): Not found: Tablet not found: 24d35379df60451998a4999c10ac45be
I20260620 03:02:18.257489 15412 leader_election.cc:302] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51a4c984db1b43ddb178038c06065720; no voters: 72aa7ecd9ca24eacb603857634cb630e, eb7608719a3c490891988f688bd61042
I20260620 03:02:18.257787 15803 raft_consensus.cc:2726] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 03:02:18.414673 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:18.413897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:40777)
0620 03:02:18.414398 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:18.414485 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":279,"threads_started":1}
I20260620 03:02:18.863180 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:18.862476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:46483)
0620 03:02:18.862938 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:18.863021 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":245,"threads_started":1}
I20260620 03:02:18.965529 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:18.964803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:42775)
0620 03:02:18.965272 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:18.965362 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":230,"threads_started":1}
I20260620 03:02:19.416296 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:19.415588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:58165)
0620 03:02:19.416008 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:19.416089 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":214,"threads_started":1}
I20260620 03:02:19.864820 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:19.864115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:42009)
0620 03:02:19.864553 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:19.864629 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":241,"threads_started":1}
I20260620 03:02:19.967283 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:19.966527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:40961)
0620 03:02:19.967012 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:19.967100 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":230,"threads_started":1}
I20260620 03:02:20.418239 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:20.417468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:37737)
0620 03:02:20.417986 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:20.418068 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":317,"threads_started":1}
I20260620 03:02:20.826530 15811 raft_consensus.cc:482] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:02:20.826740 15811 raft_consensus.cc:504] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } }
I20260620 03:02:20.827477 15811 leader_election.cc:288] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72aa7ecd9ca24eacb603857634cb630e (127.0.97.3:35411), eb7608719a3c490891988f688bd61042 (127.0.97.2:42605)
W20260620 03:02:20.828442 15412 leader_election.cc:341] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb7608719a3c490891988f688bd61042 (127.0.97.2:42605): Not found: Tablet not found: 24d35379df60451998a4999c10ac45be
W20260620 03:02:20.828639 15413 leader_election.cc:341] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72aa7ecd9ca24eacb603857634cb630e (127.0.97.3:35411): Not found: Tablet not found: 24d35379df60451998a4999c10ac45be
I20260620 03:02:20.828807 15413 leader_election.cc:302] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51a4c984db1b43ddb178038c06065720; no voters: 72aa7ecd9ca24eacb603857634cb630e, eb7608719a3c490891988f688bd61042
I20260620 03:02:20.829057 15811 raft_consensus.cc:2726] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 03:02:20.866724 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:20.865918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:56983)
0620 03:02:20.866446 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:20.866549 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":298,"threads_started":1}
I20260620 03:02:20.968986 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:20.968175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:50519)
0620 03:02:20.968681 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:20.968787 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":260,"threads_started":1}
I20260620 03:02:21.419941 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:21.419127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:40365)
0620 03:02:21.419677 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:21.419761 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":327,"threads_started":1}
I20260620 03:02:21.868248 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:21.867548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:36513)
0620 03:02:21.867986 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:21.868063 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":245,"threads_started":1}
I20260620 03:02:21.970867 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:21.970084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:55111)
0620 03:02:21.970534 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:21.970670 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":242,"threads_started":1}
I20260620 03:02:22.421788 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:22.420858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:56837)
0620 03:02:22.421424 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:22.421508 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":249,"threads_started":1}
I20260620 03:02:22.558032 15818 raft_consensus.cc:482] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:02:22.558241 15818 raft_consensus.cc:504] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } }
I20260620 03:02:22.558840 15818 leader_election.cc:288] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72aa7ecd9ca24eacb603857634cb630e (127.0.97.3:35411), eb7608719a3c490891988f688bd61042 (127.0.97.2:42605)
W20260620 03:02:22.559841 15412 leader_election.cc:341] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb7608719a3c490891988f688bd61042 (127.0.97.2:42605): Not found: Tablet not found: 24d35379df60451998a4999c10ac45be
W20260620 03:02:22.560029 15413 leader_election.cc:341] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72aa7ecd9ca24eacb603857634cb630e (127.0.97.3:35411): Not found: Tablet not found: 24d35379df60451998a4999c10ac45be
I20260620 03:02:22.560189 15413 leader_election.cc:302] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51a4c984db1b43ddb178038c06065720; no voters: 72aa7ecd9ca24eacb603857634cb630e, eb7608719a3c490891988f688bd61042
I20260620 03:02:22.560446 15818 raft_consensus.cc:2726] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 03:02:22.870223 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:22.869311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:57617)
0620 03:02:22.869939 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:22.870024 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":349,"threads_started":1}
I20260620 03:02:22.972514 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:22.971778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:53431)
0620 03:02:22.972243 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:22.972325 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":252,"threads_started":1}
I20260620 03:02:23.423442 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:23.422742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:43361)
0620 03:02:23.423191 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:23.423274 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":234,"threads_started":1}
I20260620 03:02:23.871899 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:23.871219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:36299)
0620 03:02:23.871646 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:23.871718 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":238,"threads_started":1}
I20260620 03:02:23.974208 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:23.973347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:42295)
0620 03:02:23.973914 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:23.974018 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":282,"threads_started":1}
I20260620 03:02:24.425136 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:24.424392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:41849)
0620 03:02:24.424879 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:24.424974 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":225,"threads_started":1}
I20260620 03:02:24.733902 15825 raft_consensus.cc:482] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:02:24.734112 15825 raft_consensus.cc:504] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } }
I20260620 03:02:24.734763 15825 leader_election.cc:288] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72aa7ecd9ca24eacb603857634cb630e (127.0.97.3:35411), eb7608719a3c490891988f688bd61042 (127.0.97.2:42605)
W20260620 03:02:24.735826 15412 leader_election.cc:341] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb7608719a3c490891988f688bd61042 (127.0.97.2:42605): Not found: Tablet not found: 24d35379df60451998a4999c10ac45be
W20260620 03:02:24.736007 15413 leader_election.cc:341] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72aa7ecd9ca24eacb603857634cb630e (127.0.97.3:35411): Not found: Tablet not found: 24d35379df60451998a4999c10ac45be
I20260620 03:02:24.736125 15413 leader_election.cc:302] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51a4c984db1b43ddb178038c06065720; no voters: 72aa7ecd9ca24eacb603857634cb630e, eb7608719a3c490891988f688bd61042
I20260620 03:02:24.736347 15825 raft_consensus.cc:2726] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 03:02:24.873416 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:24.872733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:36185)
0620 03:02:24.873178 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:24.873262 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":222,"threads_started":1}
I20260620 03:02:24.975798 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:24.975076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:56817)
0620 03:02:24.975510 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:24.975587 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":208,"threads_started":1}
I20260620 03:02:25.426930 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:25.426188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:47051)
0620 03:02:25.426662 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:25.426748 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":243,"threads_started":1}
I20260620 03:02:25.875334 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:25.874397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:54007)
0620 03:02:25.874917 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:25.875047 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":304,"threads_started":1}
I20260620 03:02:25.977376 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:25.976650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:53199)
0620 03:02:25.977093 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:25.977170 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":230,"threads_started":1}
I20260620 03:02:26.428658 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:26.427909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:59089)
0620 03:02:26.428409 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:26.428491 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":251,"threads_started":1}
I20260620 03:02:26.877146 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:26.876360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:34005)
0620 03:02:26.876878 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:26.876962 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":262,"threads_started":1}
I20260620 03:02:26.979004 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:26.978337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:53539)
0620 03:02:26.978765 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:26.978847 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":228,"threads_started":1}
I20260620 03:02:27.430397 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:27.429709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:43521)
0620 03:02:27.430143 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:27.430224 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":221,"threads_started":1}
I20260620 03:02:27.878767 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:27.878049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:34339)
0620 03:02:27.878506 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:27.878591 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":255,"threads_started":1}
I20260620 03:02:27.980928 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:27.979973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:44761)
0620 03:02:27.980615 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:27.980706 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":368,"threads_started":1}
I20260620 03:02:28.432067 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:28.431359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:51935)
0620 03:02:28.431821 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:28.431902 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":236,"threads_started":1}
I20260620 03:02:28.880517 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:28.879678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:42059)
0620 03:02:28.880213 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:28.880340 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":263,"threads_started":1}
I20260620 03:02:28.982599 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:28.981839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:50873)
0620 03:02:28.982301 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:28.982407 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":228,"threads_started":1}
I20260620 03:02:29.230096 15840 raft_consensus.cc:482] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:02:29.230335 15840 raft_consensus.cc:504] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa7ecd9ca24eacb603857634cb630e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35411 } } peers { permanent_uuid: "51a4c984db1b43ddb178038c06065720" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "eb7608719a3c490891988f688bd61042" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42605 } }
I20260620 03:02:29.231169 15840 leader_election.cc:288] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72aa7ecd9ca24eacb603857634cb630e (127.0.97.3:35411), eb7608719a3c490891988f688bd61042 (127.0.97.2:42605)
W20260620 03:02:29.232044 15413 leader_election.cc:341] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72aa7ecd9ca24eacb603857634cb630e (127.0.97.3:35411): Not found: Tablet not found: 24d35379df60451998a4999c10ac45be
W20260620 03:02:29.232245 15412 leader_election.cc:341] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb7608719a3c490891988f688bd61042 (127.0.97.2:42605): Not found: Tablet not found: 24d35379df60451998a4999c10ac45be
I20260620 03:02:29.232331 15412 leader_election.cc:302] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51a4c984db1b43ddb178038c06065720; no voters: 72aa7ecd9ca24eacb603857634cb630e, eb7608719a3c490891988f688bd61042
I20260620 03:02:29.232553 15840 raft_consensus.cc:2726] T 24d35379df60451998a4999c10ac45be P 51a4c984db1b43ddb178038c06065720 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 03:02:29.433951 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:29.432992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:56685)
0620 03:02:29.433618 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:29.433719 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":327,"threads_started":1}
I20260620 03:02:29.882244 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:29.881514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:35011)
0620 03:02:29.881965 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:29.882043 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":244,"threads_started":1}
I20260620 03:02:29.984478 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:29.983766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.1:58061)
0620 03:02:29.984207 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:29.984295 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":217,"threads_started":1}
I20260620 03:02:30.435811 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:30.435081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.2:54981)
0620 03:02:30.435563 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:30.435651 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":263,"threads_started":1}
I20260620 03:02:30.862416 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 15361
I20260620 03:02:30.877696 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:30.863417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47392 (local address 127.0.97.1:33399)
0620 03:02:30.864050 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:30.864057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:30.864083 (+ 26us) server_negotiation.cc:407] Connection header received
0620 03:02:30.864142 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:30.864149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:30.864375 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:30.864503 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:30.866834 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:30.867803 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:30.868980 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:30.869540 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:30.869705 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:30.872282 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:30.872299 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:30.872312 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:30.872348 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:30.875173 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:30.875790 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:30.875796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:30.875798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:30.875925 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:30.876238 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:30.876241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:30.876243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:30.876620 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:30.876831 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:30.877075 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 03:02:30.877266 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":371,"threads_started":1}
I20260620 03:02:30.878923 15521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:30.884294 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:30.883481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36449 (local address 127.0.97.3:43241)
0620 03:02:30.883972 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:30.884095 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":277,"threads_started":1}
I20260620 03:02:30.969888 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 15361
I20260620 03:02:30.988279 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 15529
I20260620 03:02:31.001267 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:30.989010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.0.97.2:42605)
0620 03:02:30.989492 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:30.989496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:30.989512 (+ 16us) server_negotiation.cc:407] Connection header received
0620 03:02:30.989566 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:30.989606 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:30.989677 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:30.989772 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:30.991250 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:30.992100 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:30.993251 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:30.993724 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:30.993819 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:30.996271 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:30.996276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:30.996279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:30.996322 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:30.999112 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:30.999707 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:30.999711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:30.999714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:30.999836 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:31.000133 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:31.000136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:31.000138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:31.000395 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:31.000578 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:31.000819 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 03:02:31.000979 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":289,"threads_started":1}
I20260620 03:02:31.002107 15652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:31.084718 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 15529
I20260620 03:02:31.102137 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 15660
I20260620 03:02:31.116371 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:31.103008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60086 (local address 127.0.97.3:35411)
0620 03:02:31.103674 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:31.103679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:31.103698 (+ 19us) server_negotiation.cc:407] Connection header received
0620 03:02:31.103753 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:31.103757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:31.103825 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:31.103920 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:31.105227 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:31.106077 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:31.107337 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:31.107741 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:31.107831 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:31.110716 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:31.110722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:31.110727 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:31.110765 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:31.113535 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:31.114338 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:31.114344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:31.114347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:31.114490 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:31.114897 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:31.114904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:31.114907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:31.115166 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:31.115460 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:31.115770 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 03:02:31.115987 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":410,"threads_started":1}
I20260620 03:02:31.117101 15783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:31.201401 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 15660
2026-06-20T03:02:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24264 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:02:31 dist-test-slave-f1j0 krb5kdc[15854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:02:31 dist-test-slave-f1j0 krb5kdc[15854](info): set up 2 sockets
Jun 20 03:02:31 dist-test-slave-f1j0 krb5kdc[15854](info): commencing operation
krb5kdc: starting...
W20260620 03:02:32.298063 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:02:32 dist-test-slave-f1j0 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:02:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:02:32.532912 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38973
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:02:32.860630 15867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:32.861012 15867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:32.861186 15867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:32.868700 15867 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 03:02:32.868857 15867 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 03:02:32.868954 15867 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 03:02:32.869058 15867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:02:32.869175 15867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:32.869302 15867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:32.869392 15867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:02:32.869473 15867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:02:32.880033 15867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:32.881551 15867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:32.882946 15867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:32.895545 15875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:32.895629 15873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:32.895591 15872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:32.899022 15867 server_base.cc:1034] running on GCE node
I20260620 03:02:32.899931 15867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:32.901355 15867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:32.902598 15867 hybrid_clock.cc:648] HybridClock initialized: now 1781924552902514 us; error 61 us; skew 500 ppm
I20260620 03:02:32.902943 15867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:32 dist-test-slave-f1j0 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:32.906164 15867 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:02:32.907619 15867 webserver.cc:460] Webserver started at http://127.0.97.62:44719/ using document root <none> and password file <none>
I20260620 03:02:32.908349 15867 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:32.908479 15867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:32.908797 15867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:02:32.911758 15867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "b717ef0d53a04f0da438426a02ac3c19"
format_stamp: "Formatted at 2026-06-20 03:02:32 on dist-test-slave-f1j0"
server_key: "e86a41db6fb6925d8d778875532d1db2"
server_key_iv: "702d2e69bb19e84e9ded96ce919dc5de"
server_key_version: "encryptionkey@0"
I20260620 03:02:32.912525 15867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "b717ef0d53a04f0da438426a02ac3c19"
format_stamp: "Formatted at 2026-06-20 03:02:32 on dist-test-slave-f1j0"
server_key: "e86a41db6fb6925d8d778875532d1db2"
server_key_iv: "702d2e69bb19e84e9ded96ce919dc5de"
server_key_version: "encryptionkey@0"
I20260620 03:02:32.917435 15867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 03:02:32.920869 15882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:32.922317 15867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:02:32.922492 15867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "b717ef0d53a04f0da438426a02ac3c19"
format_stamp: "Formatted at 2026-06-20 03:02:32 on dist-test-slave-f1j0"
server_key: "e86a41db6fb6925d8d778875532d1db2"
server_key_iv: "702d2e69bb19e84e9ded96ce919dc5de"
server_key_version: "encryptionkey@0"
I20260620 03:02:32.922710 15867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:32.936367 15867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:32.940606 15867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:32.940879 15867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:32.959059 15867 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38973
I20260620 03:02:32.959113 15933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38973 every 8 connection(s)
I20260620 03:02:32.960438 15867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:02:32.964259 15934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:02:32.965366 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 15867
I20260620 03:02:32.965637 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:02:32.966290 388 external_mini_cluster.cc:1428] Setting key c2406bf1459cb877a75da25f79073798
I20260620 03:02:32.978905 15934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19: Bootstrap starting.
Jun 20 03:02:32 dist-test-slave-f1j0 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:02:32.982545 15934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19: Neither blocks nor log segments found. Creating new log.
I20260620 03:02:32.983624 15934 log.cc:824] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:32.986464 15934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19: No bootstrap required, opened a new log
I20260620 03:02:32.989327 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:32.968263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56168 (local address 127.0.97.62:38973)
0620 03:02:32.969095 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:32.969108 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:32.969151 (+ 43us) server_negotiation.cc:407] Connection header received
0620 03:02:32.970740 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:32.970762 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:32.971164 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:32.971497 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:32.973243 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:32.974656 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:32.976726 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:32.977372 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:32.977491 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:32.982076 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:32.982084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:32.982098 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:32.982136 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:32.985193 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:32.985978 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:32.985988 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:32.986000 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:32.986162 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:32.986535 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:32.986541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:32.986544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:32.987098 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:32.987431 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:32.987853 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 03:02:32.988302 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":420,"threads_started":1}
I20260620 03:02:32.994951 15934 raft_consensus.cc:348] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b717ef0d53a04f0da438426a02ac3c19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } }
I20260620 03:02:32.995393 15934 raft_consensus.cc:374] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:02:32.995529 15934 raft_consensus.cc:729] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b717ef0d53a04f0da438426a02ac3c19, State: Initialized, Role: FOLLOWER
I20260620 03:02:32.996145 15934 consensus_queue.cc:260] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b717ef0d53a04f0da438426a02ac3c19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } }
I20260620 03:02:32.996351 15934 raft_consensus.cc:388] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:02:32.996445 15934 raft_consensus.cc:482] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:02:32.996567 15934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:02:32.998759 15934 raft_consensus.cc:504] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b717ef0d53a04f0da438426a02ac3c19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } }
I20260620 03:02:32.999313 15934 leader_election.cc:302] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b717ef0d53a04f0da438426a02ac3c19; no voters:
I20260620 03:02:32.999972 15934 leader_election.cc:288] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:02:33.000056 15939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:02:33.000823 15939 raft_consensus.cc:686] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 1 LEADER]: Becoming Leader. State: Replica: b717ef0d53a04f0da438426a02ac3c19, State: Running, Role: LEADER
I20260620 03:02:33.001309 15939 consensus_queue.cc:237] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b717ef0d53a04f0da438426a02ac3c19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } }
I20260620 03:02:33.002200 15934 sys_catalog.cc:564] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:02:33.003888 15940 sys_catalog.cc:455] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b717ef0d53a04f0da438426a02ac3c19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b717ef0d53a04f0da438426a02ac3c19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } } }
I20260620 03:02:33.004340 15940 sys_catalog.cc:458] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [sys.catalog]: This master's current role is: LEADER
I20260620 03:02:33.004220 15941 sys_catalog.cc:455] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b717ef0d53a04f0da438426a02ac3c19. Latest consensus state: current_term: 1 leader_uuid: "b717ef0d53a04f0da438426a02ac3c19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b717ef0d53a04f0da438426a02ac3c19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } } }
I20260620 03:02:33.004514 15941 sys_catalog.cc:458] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [sys.catalog]: This master's current role is: LEADER
I20260620 03:02:33.010165 15944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:02:33.016155 15944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:02:33.024433 15944 catalog_manager.cc:1347] Generated new cluster ID: fde1555b54fe4d80a7f199e742724d7f
I20260620 03:02:33.024542 15944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:02:33.045663 15944 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:02:33.046988 15944 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:02:33.059371 15944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19: Generated new TSK 0
I20260620 03:02:33.060427 15944 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:02:33.129384 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38973
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:02:33.455917 15960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:33.456203 15960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:33.456296 15960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:33.463475 15960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:33.463580 15960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:33.463696 15960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:02:33.473261 15960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:33.474682 15960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:33.475910 15960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:33.489228 15966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:33.489179 15965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:33.489261 15968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:33.490075 15960 server_base.cc:1034] running on GCE node
I20260620 03:02:33.490783 15960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:33.491756 15960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:33.492975 15960 hybrid_clock.cc:648] HybridClock initialized: now 1781924553492899 us; error 53 us; skew 500 ppm
I20260620 03:02:33.493306 15960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:33 dist-test-slave-f1j0 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:33.496418 15960 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:02:33.497889 15960 webserver.cc:460] Webserver started at http://127.0.97.1:38099/ using document root <none> and password file <none>
I20260620 03:02:33.498561 15960 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:33.498706 15960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:33.499022 15960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:02:33.501984 15960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "a39a5cebd2c947848cd4f262dac7d162"
format_stamp: "Formatted at 2026-06-20 03:02:33 on dist-test-slave-f1j0"
server_key: "db7fe6df93ebe21ae117a4d74f29e4cb"
server_key_iv: "e63a6b7b700a27b57a82ff8d23cf8d5e"
server_key_version: "encryptionkey@0"
I20260620 03:02:33.502774 15960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a39a5cebd2c947848cd4f262dac7d162"
format_stamp: "Formatted at 2026-06-20 03:02:33 on dist-test-slave-f1j0"
server_key: "db7fe6df93ebe21ae117a4d74f29e4cb"
server_key_iv: "e63a6b7b700a27b57a82ff8d23cf8d5e"
server_key_version: "encryptionkey@0"
I20260620 03:02:33.508054 15960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 03:02:33.511686 15975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:33.513093 15960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 03:02:33.513264 15960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "a39a5cebd2c947848cd4f262dac7d162"
format_stamp: "Formatted at 2026-06-20 03:02:33 on dist-test-slave-f1j0"
server_key: "db7fe6df93ebe21ae117a4d74f29e4cb"
server_key_iv: "e63a6b7b700a27b57a82ff8d23cf8d5e"
server_key_version: "encryptionkey@0"
I20260620 03:02:33.513475 15960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:33.542026 15960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:33.546048 15960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:33.546340 15960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:33.547300 15960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:02:33.548609 15960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:02:33.548713 15960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:33.548866 15960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:02:33.548940 15960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:33.582751 15960 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37765
I20260620 03:02:33.582834 16087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37765 every 8 connection(s)
I20260620 03:02:33.584044 15960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:02:33.592865 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 15960
I20260620 03:02:33.593091 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:02:33.593981 388 external_mini_cluster.cc:1428] Setting key f155ccf5b9c1c830cb3d8efd6503cee1
Jun 20 03:02:33 dist-test-slave-f1j0 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:02:33.605309 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:33.586746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39963 (local address 127.0.97.62:38973)
0620 03:02:33.587207 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:33.587213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:33.588200 (+ 987us) server_negotiation.cc:407] Connection header received
0620 03:02:33.590132 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:33.590137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:33.590212 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:33.590319 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:33.591778 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:33.592512 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:33.594945 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:33.595361 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:33.595500 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:33.598956 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:33.598960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:33.598964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:33.598997 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:33.601561 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:33.602440 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:33.602445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:33.602447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:33.602565 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:33.603118 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:33.603121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:33.603123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:33.603389 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:33.603556 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:33.604548 (+ 992us) server_negotiation.cc:299] Negotiation successful
0620 03:02:33.604823 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":222,"threads_started":1}
I20260620 03:02:33.606025 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:33.587334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:39963)
0620 03:02:33.588015 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:33.588067 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:33.589882 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:33.590526 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:33.590533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:33.591033 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:33.591630 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:33.591647 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:33.592698 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:33.592704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:33.594756 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:33.594779 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:33.595583 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:33.595591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:33.595857 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:33.596543 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:33.596565 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:33.598720 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:33.601759 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:33.601768 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:33.601787 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:33.602288 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:33.602690 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:33.602697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:33.602700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:33.602960 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:33.603551 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:33.603561 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:33.604063 (+ 502us) client_negotiation.cc:769] Sending connection context
0620 03:02:33.604372 (+ 309us) client_negotiation.cc:240] Negotiation successful
0620 03:02:33.604823 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":286,"threads_started":1}
I20260620 03:02:33.607793 16088 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38973
I20260620 03:02:33.608155 16088 heartbeater.cc:464] Registering TS with master...
I20260620 03:02:33.609045 16088 heartbeater.cc:511] Master 127.0.97.62:38973 requested a full tablet report, sending...
I20260620 03:02:33.611363 15899 ts_manager.cc:194] Registered new tserver with Master: a39a5cebd2c947848cd4f262dac7d162 (127.0.97.1:37765)
I20260620 03:02:33.612902 15899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39963
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:02:33.646811 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38973
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:02:33.972564 16094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:33.972963 16094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:33.973114 16094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:33.982153 16094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:33.982262 16094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:33.982385 16094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:02:33.992098 16094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:33.993341 16094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:33.994644 16094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:34.007737 16099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:34.007805 16100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:34.007776 16102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:34.008539 16094 server_base.cc:1034] running on GCE node
I20260620 03:02:34.009142 16094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:34.010098 16094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:34.011292 16094 hybrid_clock.cc:648] HybridClock initialized: now 1781924554011228 us; error 60 us; skew 500 ppm
I20260620 03:02:34.011631 16094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:34 dist-test-slave-f1j0 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:34.014808 16094 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:02:34.016280 16094 webserver.cc:460] Webserver started at http://127.0.97.2:42335/ using document root <none> and password file <none>
I20260620 03:02:34.016999 16094 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:34.017143 16094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:34.017458 16094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:02:34.020381 16094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "f293f78c15be405fb7883fcc06657c94"
format_stamp: "Formatted at 2026-06-20 03:02:34 on dist-test-slave-f1j0"
server_key: "c7f2e5617c9cfc4cf25da8c2f55a96ee"
server_key_iv: "d35d1aaddd17e39c6712045f290b70bd"
server_key_version: "encryptionkey@0"
I20260620 03:02:34.021117 16094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f293f78c15be405fb7883fcc06657c94"
format_stamp: "Formatted at 2026-06-20 03:02:34 on dist-test-slave-f1j0"
server_key: "c7f2e5617c9cfc4cf25da8c2f55a96ee"
server_key_iv: "d35d1aaddd17e39c6712045f290b70bd"
server_key_version: "encryptionkey@0"
I20260620 03:02:34.026543 16094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 03:02:34.030333 16109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:34.031658 16094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:02:34.031838 16094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "f293f78c15be405fb7883fcc06657c94"
format_stamp: "Formatted at 2026-06-20 03:02:34 on dist-test-slave-f1j0"
server_key: "c7f2e5617c9cfc4cf25da8c2f55a96ee"
server_key_iv: "d35d1aaddd17e39c6712045f290b70bd"
server_key_version: "encryptionkey@0"
I20260620 03:02:34.032051 16094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:34.055644 16094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:34.060093 16094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:34.060402 16094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:34.061334 16094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:02:34.062647 16094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:02:34.062754 16094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:34.062901 16094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:02:34.062981 16094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:34.095376 16094 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38031
I20260620 03:02:34.095418 16221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38031 every 8 connection(s)
I20260620 03:02:34.096691 16094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:02:34.099177 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 16094
I20260620 03:02:34.099388 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:02:34.099869 388 external_mini_cluster.cc:1428] Setting key edd8cf4b56b6d666d87782e8df70bcc4
Jun 20 03:02:34 dist-test-slave-f1j0 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:02:34.121537 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:34.099324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47843 (local address 127.0.97.62:38973)
0620 03:02:34.099522 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:34.099528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:34.100854 (+ 1326us) server_negotiation.cc:407] Connection header received
0620 03:02:34.102828 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:34.102837 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:34.102927 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:34.103062 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:34.105299 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.106069 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.108556 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.108979 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.109116 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:34.113106 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:34.113115 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:34.113123 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:34.113166 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:34.117001 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:34.118601 (+ 1600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:34.118613 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:34.118616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:34.118818 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:34.119380 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:34.119389 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:34.119395 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:34.119792 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:34.119945 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:34.120904 (+ 959us) server_negotiation.cc:299] Negotiation successful
0620 03:02:34.121147 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 03:02:34.123543 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:34.099854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:47843)
0620 03:02:34.100699 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:34.100748 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:34.102540 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:34.103499 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:34.103513 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:34.104332 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:34.105117 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:34.105143 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.106218 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.106226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:34.108390 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:34.108407 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.109231 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.109241 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:34.109593 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:34.110380 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:34.110402 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:34.112883 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:34.117178 (+ 4295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:34.117193 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:34.117215 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:34.118416 (+ 1201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:34.118975 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:34.118979 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:34.118982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:34.119218 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:34.119941 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:34.119949 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:34.120447 (+ 498us) client_negotiation.cc:769] Sending connection context
0620 03:02:34.120788 (+ 341us) client_negotiation.cc:240] Negotiation successful
0620 03:02:34.121351 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":360,"threads_started":1}
I20260620 03:02:34.125610 16222 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38973
I20260620 03:02:34.125972 16222 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 03:02:34.126788 16222 heartbeater.cc:511] Master 127.0.97.62:38973 requested a full tablet report, sending...
I20260620 03:02:34.128386 15899 ts_manager.cc:194] Registered new tserver with Master: f293f78c15be405fb7883fcc06657c94 (127.0.97.2:38031)
I20260620 03:02:34.129276 15899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:02:34.156297 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38973
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:02:34.476877 16227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:34.477207 16227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:34.477380 16227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:34.484582 16227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:34.484732 16227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:34.484916 16227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:02:34.494663 16227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:34.495995 16227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:34.497332 16227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:34.511089 16232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:34.511021 16233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:34.511068 16235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:34.512072 16227 server_base.cc:1034] running on GCE node
I20260620 03:02:34.512686 16227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:34.513813 16227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:34.515025 16227 hybrid_clock.cc:648] HybridClock initialized: now 1781924554514967 us; error 54 us; skew 500 ppm
I20260620 03:02:34.515352 16227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:34 dist-test-slave-f1j0 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:34.518493 16227 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:02:34.519923 16227 webserver.cc:460] Webserver started at http://127.0.97.3:38337/ using document root <none> and password file <none>
I20260620 03:02:34.520568 16227 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:34.520675 16227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:34.520937 16227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:02:34.523801 16227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "5250fa83d69e4193bdc59611b502fecd"
format_stamp: "Formatted at 2026-06-20 03:02:34 on dist-test-slave-f1j0"
server_key: "efb12b833224ba46f228b1373eaf074e"
server_key_iv: "4985abc43ec99407ff6d724c515685e2"
server_key_version: "encryptionkey@0"
I20260620 03:02:34.524591 16227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5250fa83d69e4193bdc59611b502fecd"
format_stamp: "Formatted at 2026-06-20 03:02:34 on dist-test-slave-f1j0"
server_key: "efb12b833224ba46f228b1373eaf074e"
server_key_iv: "4985abc43ec99407ff6d724c515685e2"
server_key_version: "encryptionkey@0"
I20260620 03:02:34.529672 16227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 03:02:34.532881 16242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:34.534332 16227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:02:34.534497 16227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "5250fa83d69e4193bdc59611b502fecd"
format_stamp: "Formatted at 2026-06-20 03:02:34 on dist-test-slave-f1j0"
server_key: "efb12b833224ba46f228b1373eaf074e"
server_key_iv: "4985abc43ec99407ff6d724c515685e2"
server_key_version: "encryptionkey@0"
I20260620 03:02:34.534693 16227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:34.559289 16227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:34.563745 16227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:34.564044 16227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:34.565083 16227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:02:34.566398 16227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:02:34.566471 16227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:34.566641 16227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:02:34.566702 16227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:34.599735 16227 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37821
I20260620 03:02:34.599771 16354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37821 every 8 connection(s)
I20260620 03:02:34.601023 16227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:02:34.608233 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 16227
I20260620 03:02:34.608424 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:02:34.608883 388 external_mini_cluster.cc:1428] Setting key c59b01a9180e906cd8029b1d14852d64
Jun 20 03:02:34 dist-test-slave-f1j0 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:02:34.617225 16088 heartbeater.cc:503] Master 127.0.97.62:38973 was elected leader, sending a full tablet report...
I20260620 03:02:34.621109 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:34.603712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53007 (local address 127.0.97.62:38973)
0620 03:02:34.603935 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:34.603941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:34.605039 (+ 1098us) server_negotiation.cc:407] Connection header received
0620 03:02:34.606863 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:34.606867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:34.606937 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:34.607024 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:34.608471 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.609352 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.611189 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.611691 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.611814 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:34.615181 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:34.615186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:34.615189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:34.615231 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:34.618057 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:34.618912 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:34.618916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:34.618918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:34.619043 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:34.619534 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:34.619538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:34.619540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:34.619834 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:34.619948 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:34.620694 (+ 746us) server_negotiation.cc:299] Negotiation successful
0620 03:02:34.620862 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 03:02:34.622228 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:34.604159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:53007)
0620 03:02:34.604870 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:34.604913 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:34.606614 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:34.607235 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:34.607242 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:34.607730 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:34.608284 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:34.608333 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.609505 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.609511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:34.611045 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:34.611057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.611978 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.611988 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:34.612193 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:34.612836 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:34.612858 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:34.614998 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:34.618186 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:34.618194 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:34.618210 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:34.618789 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:34.619197 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:34.619201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:34.619204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:34.619391 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:34.619973 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:34.619979 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:34.620322 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 03:02:34.620551 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 03:02:34.620940 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":289,"threads_started":1}
I20260620 03:02:34.623612 16355 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38973
I20260620 03:02:34.623950 16355 heartbeater.cc:464] Registering TS with master...
I20260620 03:02:34.624830 16355 heartbeater.cc:511] Master 127.0.97.62:38973 requested a full tablet report, sending...
I20260620 03:02:34.626288 15899 ts_manager.cc:194] Registered new tserver with Master: 5250fa83d69e4193bdc59611b502fecd (127.0.97.3:37821)
I20260620 03:02:34.627051 15899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53007
I20260620 03:02:34.636899 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:02:34.654663 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:34.641719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44166 (local address 127.0.97.62:38973)
0620 03:02:34.641979 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:34.641987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:34.642390 (+ 403us) server_negotiation.cc:407] Connection header received
0620 03:02:34.642617 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:34.642627 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:34.642696 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:34.642820 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:34.644242 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.644992 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.646387 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.646798 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.646945 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:34.649427 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:34.649432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:34.649435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:34.649469 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:34.651928 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:34.652644 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:34.652657 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:34.652659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:34.652778 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:34.653141 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:34.653148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:34.653151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:34.653484 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:34.653761 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:34.654105 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 03:02:34.654343 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 03:02:34.658677 15899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:02:34.665401 15899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:02:34.702152 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:34.678375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44538 (local address 127.0.97.3:37821)
0620 03:02:34.692525 (+ 14150us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:34.692539 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:34.694269 (+ 1730us) server_negotiation.cc:407] Connection header received
0620 03:02:34.694381 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:34.694388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:34.694624 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:34.694800 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:34.695821 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.697066 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.699463 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.700599 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.700733 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:34.700977 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:34.701606 (+ 629us) server_negotiation.cc:299] Negotiation successful
0620 03:02:34.701783 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13891,"thread_start_us":14159,"threads_started":1}
I20260620 03:02:34.702555 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:34.677863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.0.97.1:37765)
0620 03:02:34.689608 (+ 11745us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:34.689621 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:34.693563 (+ 3942us) server_negotiation.cc:407] Connection header received
0620 03:02:34.693715 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:34.693723 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:34.694018 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:34.694227 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:34.695573 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.697214 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.700175 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.701536 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.701678 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:34.701893 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:34.702082 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 03:02:34.702237 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11489,"thread_start_us":11281,"threads_started":1}
I20260620 03:02:34.702628 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:34.677141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37765 (local address 127.0.0.1:44948)
0620 03:02:34.693228 (+ 16087us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:34.693254 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:34.693422 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:34.694506 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:34.694514 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:34.694563 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:34.695165 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:34.695186 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.697394 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.697401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:34.700008 (+ 2607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:34.700022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.701729 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.701735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:34.701909 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:34.701931 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 03:02:34.702036 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 03:02:34.702195 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15944,"mutex_wait_us":207,"spinlock_wait_cycles":8320,"thread_start_us":15795,"threads_started":1}
I20260620 03:02:34.702967 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:34.677664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37821 (local address 127.0.0.1:44538)
0620 03:02:34.693994 (+ 16330us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:34.694010 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:34.694136 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:34.694916 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:34.694925 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:34.694948 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:34.695636 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:34.695652 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.697255 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.697262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:34.699294 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:34.699315 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.700811 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.700817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:34.701003 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:34.701066 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 03:02:34.701211 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 03:02:34.701368 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16193,"mutex_wait_us":1,"thread_start_us":8907,"threads_started":1}
I20260620 03:02:34.705041 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:34.677030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38031 (local address 127.0.0.1:46662)
0620 03:02:34.686994 (+ 9964us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:34.687038 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:34.687258 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:34.694506 (+ 7248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:34.694514 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:34.694564 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:34.695554 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:34.695567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.697489 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.697496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:34.700011 (+ 2515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:34.700022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.704301 (+ 4279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.704305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:34.704412 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:34.704433 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 03:02:34.704542 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 03:02:34.704666 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9699,"mutex_wait_us":312,"spinlock_wait_cycles":5632,"thread_start_us":16692,"threads_started":1}
I20260620 03:02:34.705067 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:34.677314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46662 (local address 127.0.97.2:38031)
0620 03:02:34.693644 (+ 16330us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:34.693653 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:34.693676 (+ 23us) server_negotiation.cc:407] Connection header received
0620 03:02:34.693762 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:34.693767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:34.694005 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:34.694233 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:34.695862 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.697343 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.701977 (+ 4634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:34.703885 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:34.704019 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:34.704191 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:34.704585 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 03:02:34.704757 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16133,"thread_start_us":15996,"threads_started":1}
I20260620 03:02:34.707082 16290 tablet_service.cc:1467] Processing CreateTablet for tablet e683fff446564ea8978349c1afda3e85 (DEFAULT_TABLE table=test-table [id=f5529fc648474b9f93810c4af27593a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:02:34.708683 16290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e683fff446564ea8978349c1afda3e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:02:34.708429 16157 tablet_service.cc:1467] Processing CreateTablet for tablet e683fff446564ea8978349c1afda3e85 (DEFAULT_TABLE table=test-table [id=f5529fc648474b9f93810c4af27593a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:02:34.709808 16157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e683fff446564ea8978349c1afda3e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:02:34.710186 16023 tablet_service.cc:1467] Processing CreateTablet for tablet e683fff446564ea8978349c1afda3e85 (DEFAULT_TABLE table=test-table [id=f5529fc648474b9f93810c4af27593a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:02:34.711318 16023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e683fff446564ea8978349c1afda3e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:02:34.722395 16373 tablet_bootstrap.cc:492] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162: Bootstrap starting.
I20260620 03:02:34.724608 16375 tablet_bootstrap.cc:492] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd: Bootstrap starting.
I20260620 03:02:34.726433 16373 tablet_bootstrap.cc:654] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162: Neither blocks nor log segments found. Creating new log.
I20260620 03:02:34.728084 16375 tablet_bootstrap.cc:654] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd: Neither blocks nor log segments found. Creating new log.
I20260620 03:02:34.728230 16373 log.cc:824] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:34.729288 16375 log.cc:824] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:34.729436 16374 tablet_bootstrap.cc:492] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94: Bootstrap starting.
I20260620 03:02:34.731487 16373 tablet_bootstrap.cc:492] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162: No bootstrap required, opened a new log
I20260620 03:02:34.731793 16373 ts_tablet_manager.cc:1397] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 03:02:34.732417 16375 tablet_bootstrap.cc:492] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd: No bootstrap required, opened a new log
I20260620 03:02:34.732677 16375 ts_tablet_manager.cc:1397] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 03:02:34.732719 16374 tablet_bootstrap.cc:654] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94: Neither blocks nor log segments found. Creating new log.
I20260620 03:02:34.733748 16374 log.cc:824] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:34.736768 16374 tablet_bootstrap.cc:492] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94: No bootstrap required, opened a new log
I20260620 03:02:34.737039 16374 ts_tablet_manager.cc:1397] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 03:02:34.742700 16373 raft_consensus.cc:348] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } }
I20260620 03:02:34.743187 16373 raft_consensus.cc:374] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:02:34.743296 16373 raft_consensus.cc:729] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39a5cebd2c947848cd4f262dac7d162, State: Initialized, Role: FOLLOWER
I20260620 03:02:34.743288 16375 raft_consensus.cc:348] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } }
I20260620 03:02:34.743693 16375 raft_consensus.cc:374] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:02:34.743791 16375 raft_consensus.cc:729] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5250fa83d69e4193bdc59611b502fecd, State: Initialized, Role: FOLLOWER
I20260620 03:02:34.743956 16373 consensus_queue.cc:260] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } }
I20260620 03:02:34.744354 16375 consensus_queue.cc:260] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } }
I20260620 03:02:34.745995 16373 ts_tablet_manager.cc:1428] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260620 03:02:34.745440 16374 raft_consensus.cc:348] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } }
I20260620 03:02:34.747112 16374 raft_consensus.cc:374] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:02:34.747187 16355 heartbeater.cc:503] Master 127.0.97.62:38973 was elected leader, sending a full tablet report...
I20260620 03:02:34.747205 16374 raft_consensus.cc:729] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f293f78c15be405fb7883fcc06657c94, State: Initialized, Role: FOLLOWER
I20260620 03:02:34.747851 16375 ts_tablet_manager.cc:1428] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260620 03:02:34.747840 16374 consensus_queue.cc:260] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } }
I20260620 03:02:34.749773 16222 heartbeater.cc:503] Master 127.0.97.62:38973 was elected leader, sending a full tablet report...
I20260620 03:02:34.750396 16374 ts_tablet_manager.cc:1428] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260620 03:02:34.836493 16089 tablet.cc:2369] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:02:34.848716 16223 tablet.cc:2369] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:02:34.852691 16356 tablet.cc:2369] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:02:35.017796 16379 raft_consensus.cc:482] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:02:35.018221 16379 raft_consensus.cc:504] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } }
I20260620 03:02:35.020068 16379 leader_election.cc:288] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f293f78c15be405fb7883fcc06657c94 (127.0.97.2:38031), 5250fa83d69e4193bdc59611b502fecd (127.0.97.3:37821)
I20260620 03:02:35.029606 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:35.020599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60733 (local address 127.0.97.2:38031)
0620 03:02:35.020832 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:35.020840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:35.021503 (+ 663us) server_negotiation.cc:407] Connection header received
0620 03:02:35.021853 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:35.021858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:35.021953 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:35.022052 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:35.022826 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:35.024146 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:35.027030 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:35.028275 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:35.028414 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:35.028516 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:35.029198 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 03:02:35.029328 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 03:02:35.029606 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:35.020536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59791 (local address 127.0.97.3:37821)
0620 03:02:35.020758 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:35.020763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:35.021872 (+ 1109us) server_negotiation.cc:407] Connection header received
0620 03:02:35.022053 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:35.022059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:35.022143 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:35.022236 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:35.023222 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:35.024267 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:35.027030 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:35.028286 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:35.028413 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:35.028520 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:35.029198 (+ 678us) server_negotiation.cc:299] Negotiation successful
0620 03:02:35.029332 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 03:02:35.029801 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:35.020623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37821 (local address 127.0.97.1:59791)
0620 03:02:35.021778 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:35.021793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:35.021898 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:35.022383 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:35.022390 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:35.022417 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:35.022984 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:35.023000 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:35.024404 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:35.024409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:35.026863 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:35.026877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:35.028592 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:35.028597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:35.028888 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:35.029030 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 03:02:35.029149 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 03:02:35.029288 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"mutex_wait_us":469,"thread_start_us":576,"threads_started":1}
I20260620 03:02:35.030119 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:35.020623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38031 (local address 127.0.97.1:60733)
0620 03:02:35.021373 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:35.021391 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:35.021669 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:35.022058 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:35.022063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:35.022090 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:35.022588 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:35.022601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:35.024281 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:35.024285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:35.026866 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:35.026877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:35.028588 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:35.028595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:35.028887 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:35.029031 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 03:02:35.029149 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 03:02:35.029295 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":176,"thread_start_us":373,"threads_started":1}
I20260620 03:02:35.030853 16310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e683fff446564ea8978349c1afda3e85" candidate_uuid: "a39a5cebd2c947848cd4f262dac7d162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5250fa83d69e4193bdc59611b502fecd" is_pre_election: true
I20260620 03:02:35.031225 16177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e683fff446564ea8978349c1afda3e85" candidate_uuid: "a39a5cebd2c947848cd4f262dac7d162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f293f78c15be405fb7883fcc06657c94" is_pre_election: true
I20260620 03:02:35.031483 16310 raft_consensus.cc:2454] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a39a5cebd2c947848cd4f262dac7d162 in term 0.
I20260620 03:02:35.031766 16177 raft_consensus.cc:2454] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a39a5cebd2c947848cd4f262dac7d162 in term 0.
I20260620 03:02:35.032366 15979 leader_election.cc:302] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5250fa83d69e4193bdc59611b502fecd, a39a5cebd2c947848cd4f262dac7d162; no voters:
I20260620 03:02:35.032810 16379 raft_consensus.cc:2781] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:02:35.032960 16379 raft_consensus.cc:482] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:02:35.033067 16379 raft_consensus.cc:3037] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:02:35.035437 16379 raft_consensus.cc:504] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } }
I20260620 03:02:35.036123 16379 leader_election.cc:288] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [CANDIDATE]: Term 1 election: Requested vote from peers f293f78c15be405fb7883fcc06657c94 (127.0.97.2:38031), 5250fa83d69e4193bdc59611b502fecd (127.0.97.3:37821)
I20260620 03:02:35.036549 16310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e683fff446564ea8978349c1afda3e85" candidate_uuid: "a39a5cebd2c947848cd4f262dac7d162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5250fa83d69e4193bdc59611b502fecd"
I20260620 03:02:35.036535 16177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e683fff446564ea8978349c1afda3e85" candidate_uuid: "a39a5cebd2c947848cd4f262dac7d162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f293f78c15be405fb7883fcc06657c94"
I20260620 03:02:35.036752 16310 raft_consensus.cc:3037] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:02:35.036790 16177 raft_consensus.cc:3037] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:02:35.039211 16177 raft_consensus.cc:2454] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39a5cebd2c947848cd4f262dac7d162 in term 1.
I20260620 03:02:35.039249 16310 raft_consensus.cc:2454] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39a5cebd2c947848cd4f262dac7d162 in term 1.
I20260620 03:02:35.039629 15979 leader_election.cc:302] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5250fa83d69e4193bdc59611b502fecd, a39a5cebd2c947848cd4f262dac7d162; no voters:
I20260620 03:02:35.039922 16379 raft_consensus.cc:2781] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:02:35.040514 16379 raft_consensus.cc:686] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [term 1 LEADER]: Becoming Leader. State: Replica: a39a5cebd2c947848cd4f262dac7d162, State: Running, Role: LEADER
I20260620 03:02:35.041086 16379 consensus_queue.cc:237] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } }
I20260620 03:02:35.045473 15899 catalog_manager.cc:5495] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 reported cstate change: term changed from 0 to 1, leader changed from <none> to a39a5cebd2c947848cd4f262dac7d162 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a39a5cebd2c947848cd4f262dac7d162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } health_report { overall_health: UNKNOWN } } }
I20260620 03:02:35.071928 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:35.067074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44960 (local address 127.0.97.1:37765)
0620 03:02:35.067248 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:35.067252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:35.067269 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:02:35.067352 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:35.067356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:35.067452 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:35.067569 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:02:35.067902 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:35.068610 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:35.070099 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:35.070489 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:35.070583 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:35.070990 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:02:35.071475 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:02:35.071581 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:35.071667 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 03:02:35.071736 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 03:02:35.079728 16310 raft_consensus.cc:1261] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd [term 1 FOLLOWER]: Refusing update from remote peer a39a5cebd2c947848cd4f262dac7d162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:02:35.079730 16177 raft_consensus.cc:1261] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94 [term 1 FOLLOWER]: Refusing update from remote peer a39a5cebd2c947848cd4f262dac7d162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:02:35.080660 16384 consensus_queue.cc:1035] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:02:35.081022 16379 consensus_queue.cc:1035] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:02:35.098821 16387 mvcc.cc:205] Tried to move back new op lower bound from 7298762977597079552 to 7298762977452167168. Current Snapshot: MvccSnapshot[applied={T|T < 7298762977597079552 or (T in {7298762977597079552})}]
I20260620 03:02:35.109531 15899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:02:35.109984 15899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:02:35.114897 15899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19: Sending DeleteTablet for 3 replicas of tablet e683fff446564ea8978349c1afda3e85
I20260620 03:02:35.116937 16023 tablet_service.cc:1514] Processing DeleteTablet for tablet e683fff446564ea8978349c1afda3e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:02:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44948
I20260620 03:02:35.117676 16157 tablet_service.cc:1514] Processing DeleteTablet for tablet e683fff446564ea8978349c1afda3e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:02:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46662
I20260620 03:02:35.119669 16290 tablet_service.cc:1514] Processing DeleteTablet for tablet e683fff446564ea8978349c1afda3e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:02:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44538
I20260620 03:02:35.120121 16398 tablet_replica.cc:330] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94: stopping tablet replica
I20260620 03:02:35.120784 16398 raft_consensus.cc:2229] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:02:35.121241 16398 raft_consensus.cc:2258] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:02:35.121477 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 15960
I20260620 03:02:35.122349 16289 tablet_service.cc:1514] Processing DeleteTablet for tablet e683fff446564ea8978349c1afda3e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:02:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44538
I20260620 03:02:35.125629 16399 tablet_replica.cc:330] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd: stopping tablet replica
I20260620 03:02:35.126039 16399 raft_consensus.cc:2229] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:02:35.126492 16399 raft_consensus.cc:2258] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:02:35.127198 16398 ts_tablet_manager.cc:1905] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:02:35.133976 16397 tablet_replica.cc:330] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162: stopping tablet replica
I20260620 03:02:35.134411 16397 raft_consensus.cc:2229] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:02:35.134812 16399 ts_tablet_manager.cc:1905] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:02:35.135136 16397 raft_consensus.cc:2258] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:02:35.137804 16397 ts_tablet_manager.cc:1905] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:02:35.139292 16398 ts_tablet_manager.cc:1918] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:02:35.139528 16398 log.cc:1196] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/e683fff446564ea8978349c1afda3e85
I20260620 03:02:35.140074 16398 ts_tablet_manager.cc:1939] T e683fff446564ea8978349c1afda3e85 P f293f78c15be405fb7883fcc06657c94: Deleting consensus metadata
I20260620 03:02:35.141922 15883 catalog_manager.cc:4841] TS f293f78c15be405fb7883fcc06657c94 (127.0.97.2:38031): tablet e683fff446564ea8978349c1afda3e85 (table test-table [id=f5529fc648474b9f93810c4af27593a1]) successfully deleted
I20260620 03:02:35.144785 16399 ts_tablet_manager.cc:1918] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:02:35.144954 16399 log.cc:1196] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/e683fff446564ea8978349c1afda3e85
I20260620 03:02:35.145449 16399 ts_tablet_manager.cc:1939] T e683fff446564ea8978349c1afda3e85 P 5250fa83d69e4193bdc59611b502fecd: Deleting consensus metadata
I20260620 03:02:35.146689 16397 ts_tablet_manager.cc:1918] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:02:35.147188 15886 catalog_manager.cc:4841] TS 5250fa83d69e4193bdc59611b502fecd (127.0.97.3:37821): tablet e683fff446564ea8978349c1afda3e85 (table test-table [id=f5529fc648474b9f93810c4af27593a1]) successfully deleted
I20260620 03:02:35.146917 16397 log.cc:1196] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/e683fff446564ea8978349c1afda3e85
W20260620 03:02:35.147840 15886 catalog_manager.cc:4808] TS 5250fa83d69e4193bdc59611b502fecd (127.0.97.3:37821): delete failed for tablet e683fff446564ea8978349c1afda3e85 because the tablet was not found. No further retry: Not found: Tablet not found: e683fff446564ea8978349c1afda3e85
I20260620 03:02:35.148043 16397 ts_tablet_manager.cc:1939] T e683fff446564ea8978349c1afda3e85 P a39a5cebd2c947848cd4f262dac7d162: Deleting consensus metadata
I20260620 03:02:35.149706 15884 catalog_manager.cc:4841] TS a39a5cebd2c947848cd4f262dac7d162 (127.0.97.1:37765): tablet e683fff446564ea8978349c1afda3e85 (table test-table [id=f5529fc648474b9f93810c4af27593a1]) successfully deleted
Jun 20 03:02:35 dist-test-slave-f1j0 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:02:35.159116 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:35.123972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44966 (local address 127.0.97.1:37765)
0620 03:02:35.124198 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:35.124207 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:35.138468 (+ 14261us) server_negotiation.cc:407] Connection header received
0620 03:02:35.138679 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:35.138685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:35.138762 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:35.138872 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:35.144068 (+ 5196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:35.145398 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:35.147507 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:35.148179 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:35.148336 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:35.153071 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:35.153082 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:35.153096 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:35.153133 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:35.156205 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:35.156959 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:35.156967 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:35.156969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:35.157111 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:35.157502 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:35.157506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:35.157508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:35.157903 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:35.158213 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:35.158528 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 03:02:35.158759 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 03:02:35.159833 16083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:35.259020 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 15960
I20260620 03:02:35.277994 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 16094
Jun 20 03:02:35 dist-test-slave-f1j0 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:02:35.292397 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:35.278768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46678 (local address 127.0.97.2:38031)
0620 03:02:35.278929 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:35.278933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:35.278950 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:02:35.279072 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:35.279076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:35.279138 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:35.279251 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:35.280446 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:35.281147 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:35.282487 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:35.282871 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:35.282961 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:35.287056 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:35.287069 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:35.287085 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:35.287116 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:35.289956 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:35.290631 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:35.290639 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:35.290641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:35.290771 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:35.291094 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:35.291097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:35.291099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:35.291459 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:35.291672 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:35.291906 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 03:02:35.292067 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 03:02:35.293048 16217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:35.382139 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 16094
I20260620 03:02:35.399228 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 16227
Jun 20 03:02:35 dist-test-slave-f1j0 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:02:35.415027 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:35.400023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44552 (local address 127.0.97.3:37821)
0620 03:02:35.400269 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:35.400274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:35.400299 (+ 25us) server_negotiation.cc:407] Connection header received
0620 03:02:35.400362 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:35.400366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:35.400436 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:35.400539 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:35.401915 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:35.402688 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:35.404009 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:35.404403 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:35.404538 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:35.409254 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:35.409271 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:35.409290 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:35.409332 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:35.412395 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:35.413148 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:35.413156 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:35.413158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:35.413310 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:35.413701 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:35.413704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:35.413706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:35.414096 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:35.414324 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:35.414596 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 03:02:35.414778 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260620 03:02:35.415686 16350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:35.504220 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 16227
I20260620 03:02:35.520851 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 15867
I20260620 03:02:35.522055 15929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:35.633873 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 15867
I20260620 03:02:55.655812 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38973
--webserver_interface=127.0.97.62
--webserver_port=44719
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:02:55.989727 16405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:55.990047 16405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:55.990208 16405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:55.997622 16405 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 03:02:55.997730 16405 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 03:02:55.997829 16405 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 03:02:55.997925 16405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:02:55.998018 16405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:55.998137 16405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:55.998222 16405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:02:55.998301 16405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:02:56.008291 16405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:56.009732 16405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:56.011112 16405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:56.033864 16414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:56.033986 16411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:56.033869 16412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:56.034490 16405 server_base.cc:1034] running on GCE node
I20260620 03:02:56.035801 16405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:56.037456 16405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:56.038705 16405 hybrid_clock.cc:648] HybridClock initialized: now 1781924576038634 us; error 84 us; skew 500 ppm
I20260620 03:02:56.039124 16405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:56 dist-test-slave-f1j0 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:56.045519 16405 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:02:56.047290 16405 webserver.cc:460] Webserver started at http://127.0.97.62:44719/ using document root <none> and password file <none>
I20260620 03:02:56.048168 16405 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:56.048321 16405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:56.054510 16405 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 03:02:56.057687 16421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:56.059142 16405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:02:56.059350 16405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "b717ef0d53a04f0da438426a02ac3c19"
format_stamp: "Formatted at 2026-06-20 03:02:32 on dist-test-slave-f1j0"
server_key: "e86a41db6fb6925d8d778875532d1db2"
server_key_iv: "702d2e69bb19e84e9ded96ce919dc5de"
server_key_version: "encryptionkey@0"
I20260620 03:02:56.060479 16405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:56.096733 16405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:56.101640 16405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:56.101979 16405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:56.125792 16405 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38973
I20260620 03:02:56.125813 16472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38973 every 8 connection(s)
I20260620 03:02:56.127826 16405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:02:56.129748 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 16405
I20260620 03:02:56.130795 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:37765
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38973
--webserver_port=38099
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:02:56.139974 16473 sys_catalog.cc:263] Verifying existing consensus state
I20260620 03:02:56.143981 16473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19: Bootstrap starting.
I20260620 03:02:56.164048 16473 log.cc:824] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:56.175958 16473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 03:02:56.176548 16473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19: Bootstrap complete.
I20260620 03:02:56.186146 16473 raft_consensus.cc:348] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b717ef0d53a04f0da438426a02ac3c19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } }
I20260620 03:02:56.187016 16473 raft_consensus.cc:729] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b717ef0d53a04f0da438426a02ac3c19, State: Initialized, Role: FOLLOWER
I20260620 03:02:56.187726 16473 consensus_queue.cc:260] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b717ef0d53a04f0da438426a02ac3c19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } }
I20260620 03:02:56.187970 16473 raft_consensus.cc:388] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:02:56.188097 16473 raft_consensus.cc:482] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:02:56.188292 16473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 1 FOLLOWER]: Advancing to term 2
I20260620 03:02:56.192288 16473 raft_consensus.cc:504] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b717ef0d53a04f0da438426a02ac3c19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } }
I20260620 03:02:56.192818 16473 leader_election.cc:302] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b717ef0d53a04f0da438426a02ac3c19; no voters:
I20260620 03:02:56.193529 16473 leader_election.cc:288] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 03:02:56.193698 16477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 03:02:56.194754 16477 raft_consensus.cc:686] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 2 LEADER]: Becoming Leader. State: Replica: b717ef0d53a04f0da438426a02ac3c19, State: Running, Role: LEADER
I20260620 03:02:56.195370 16477 consensus_queue.cc:237] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b717ef0d53a04f0da438426a02ac3c19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } }
I20260620 03:02:56.195765 16473 sys_catalog.cc:564] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:02:56.197794 16479 sys_catalog.cc:455] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b717ef0d53a04f0da438426a02ac3c19. Latest consensus state: current_term: 2 leader_uuid: "b717ef0d53a04f0da438426a02ac3c19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b717ef0d53a04f0da438426a02ac3c19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } } }
I20260620 03:02:56.198199 16479 sys_catalog.cc:458] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [sys.catalog]: This master's current role is: LEADER
I20260620 03:02:56.198773 16478 sys_catalog.cc:455] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b717ef0d53a04f0da438426a02ac3c19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b717ef0d53a04f0da438426a02ac3c19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } } }
I20260620 03:02:56.199071 16478 sys_catalog.cc:458] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [sys.catalog]: This master's current role is: LEADER
I20260620 03:02:56.206583 16483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:02:56.215036 16483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:02:56.216566 16483 catalog_manager.cc:1259] Loaded cluster ID: fde1555b54fe4d80a7f199e742724d7f
I20260620 03:02:56.216727 16483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:02:56.221719 16483 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:02:56.223280 16483 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19: Loaded TSK: 0
I20260620 03:02:56.258636 16483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19: Generated new TSK 1
I20260620 03:02:56.259052 16483 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 03:02:56.488781 16475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:56.489096 16475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:56.489279 16475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:56.497044 16475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:56.497162 16475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:56.497336 16475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:02:56.507581 16475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:37765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38099
--tserver_master_addrs=127.0.97.62:38973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:56.508973 16475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:56.510704 16475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:56.525403 16501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:56.525403 16500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:56.525714 16503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:56.526966 16475 server_base.cc:1034] running on GCE node
I20260620 03:02:56.527595 16475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:56.528640 16475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:56.529884 16475 hybrid_clock.cc:648] HybridClock initialized: now 1781924576529810 us; error 67 us; skew 500 ppm
I20260620 03:02:56.530225 16475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:56 dist-test-slave-f1j0 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:56.533644 16475 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:02:56.535223 16475 webserver.cc:460] Webserver started at http://127.0.97.1:38099/ using document root <none> and password file <none>
I20260620 03:02:56.535956 16475 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:56.536111 16475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:56.541738 16475 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 03:02:56.544560 16510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:56.546025 16475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:02:56.546198 16475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "a39a5cebd2c947848cd4f262dac7d162"
format_stamp: "Formatted at 2026-06-20 03:02:33 on dist-test-slave-f1j0"
server_key: "db7fe6df93ebe21ae117a4d74f29e4cb"
server_key_iv: "e63a6b7b700a27b57a82ff8d23cf8d5e"
server_key_version: "encryptionkey@0"
I20260620 03:02:56.547179 16475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:56.564994 16475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:56.569388 16475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:56.569710 16475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:56.570749 16475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:02:56.572129 16475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:02:56.572263 16475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:56.572391 16475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:02:56.572479 16475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:56.605677 16475 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37765
I20260620 03:02:56.605748 16622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37765 every 8 connection(s)
I20260620 03:02:56.607321 16475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:02:56.608469 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 16475
I20260620 03:02:56.609115 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38031
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38973
--webserver_port=42335
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:02:56 dist-test-slave-f1j0 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:02:56.634790 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:56.610590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:50155)
0620 03:02:56.611488 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:56.611548 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:56.614282 (+ 2734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:56.615699 (+ 1417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:56.615713 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:56.616524 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:56.617342 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:56.617363 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:56.619523 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:56.619531 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:56.621770 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:56.621789 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:56.622873 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:56.622881 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:56.623203 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:56.623930 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:56.623965 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:56.626693 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:56.630432 (+ 3739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:56.630441 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:56.630463 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:56.630881 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:56.631322 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:56.631328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:56.631332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:56.631530 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:56.632262 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:56.632270 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:56.632616 (+ 346us) client_negotiation.cc:769] Sending connection context
0620 03:02:56.632874 (+ 258us) client_negotiation.cc:240] Negotiation successful
0620 03:02:56.633431 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":338,"threads_started":1}
I20260620 03:02:56.634790 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:56.610418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50155 (local address 127.0.97.62:38973)
0620 03:02:56.611123 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:56.611139 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:56.611700 (+ 561us) server_negotiation.cc:407] Connection header received
0620 03:02:56.614823 (+ 3123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:56.614837 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:56.615180 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:56.615490 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:56.617546 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:56.619313 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:56.621934 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:56.622614 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:56.622749 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:56.626987 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:56.626995 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:56.627012 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:56.627052 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:56.630279 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:56.631037 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:56.631044 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:56.631051 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:56.631207 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:56.631653 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:56.631662 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:56.631665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:56.632124 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:56.632334 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:56.633094 (+ 760us) server_negotiation.cc:299] Negotiation successful
0620 03:02:56.633468 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":279,"threads_started":1}
I20260620 03:02:56.637928 16623 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38973
I20260620 03:02:56.638240 16623 heartbeater.cc:464] Registering TS with master...
I20260620 03:02:56.639050 16623 heartbeater.cc:511] Master 127.0.97.62:38973 requested a full tablet report, sending...
I20260620 03:02:56.641237 16438 ts_manager.cc:194] Registered new tserver with Master: a39a5cebd2c947848cd4f262dac7d162 (127.0.97.1:37765)
I20260620 03:02:56.642818 16438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50155
W20260620 03:02:56.947639 16627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:56.947968 16627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:56.948137 16627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:56.955667 16627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:56.955811 16627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:56.955996 16627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:02:56.966737 16627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42335
--tserver_master_addrs=127.0.97.62:38973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:56.968093 16627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:56.969437 16627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:56.982472 16633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:56.982597 16635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:56.982618 16632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:56.983754 16627 server_base.cc:1034] running on GCE node
I20260620 03:02:56.984330 16627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:56.985237 16627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:56.986428 16627 hybrid_clock.cc:648] HybridClock initialized: now 1781924576986389 us; error 29 us; skew 500 ppm
I20260620 03:02:56.986704 16627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:56 dist-test-slave-f1j0 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:56.989692 16627 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:02:56.991161 16627 webserver.cc:460] Webserver started at http://127.0.97.2:42335/ using document root <none> and password file <none>
I20260620 03:02:56.991864 16627 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:56.992033 16627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:56.997560 16627 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 03:02:57.000311 16642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:57.001744 16627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:02:57.001932 16627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "f293f78c15be405fb7883fcc06657c94"
format_stamp: "Formatted at 2026-06-20 03:02:34 on dist-test-slave-f1j0"
server_key: "c7f2e5617c9cfc4cf25da8c2f55a96ee"
server_key_iv: "d35d1aaddd17e39c6712045f290b70bd"
server_key_version: "encryptionkey@0"
I20260620 03:02:57.002879 16627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:57.023571 16627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:57.027787 16627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:57.028090 16627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:57.029150 16627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:02:57.030689 16627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:02:57.030818 16627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:57.030972 16627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:02:57.031061 16627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:57.065022 16627 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38031
I20260620 03:02:57.065110 16754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38031 every 8 connection(s)
I20260620 03:02:57.066594 16627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:02:57.067883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 16627
I20260620 03:02:57.068655 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37821
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38973
--webserver_port=38337
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 03:02:57 dist-test-slave-f1j0 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:02:57.087257 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:57.069486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41331 (local address 127.0.97.62:38973)
0620 03:02:57.069699 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:57.069704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:57.071238 (+ 1534us) server_negotiation.cc:407] Connection header received
0620 03:02:57.073037 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:57.073043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:57.073130 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:57.073249 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:57.074825 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.075538 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.077094 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.077518 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.077689 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:57.081603 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:57.081614 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:57.081619 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:57.081661 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:57.084199 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:57.084927 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:57.084939 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:57.084941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:57.085055 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:57.085474 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:57.085481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:57.085484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:57.085835 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:57.085994 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:57.086677 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 03:02:57.086923 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 03:02:57.088241 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:57.070278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:41331)
0620 03:02:57.071071 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:57.071107 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:57.072742 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:57.073492 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:57.073500 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:57.074090 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:57.074637 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:57.074655 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.075712 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.075725 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:57.076945 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:57.076957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.077786 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.077793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:57.078017 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:57.078723 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:57.078750 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:57.081292 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:57.084332 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:57.084341 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:57.084357 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:57.084793 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:57.085168 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:57.085173 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:57.085177 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:57.085355 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:57.085961 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:57.085969 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:57.086317 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 03:02:57.086604 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 03:02:57.087127 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":338,"threads_started":1}
I20260620 03:02:57.090029 16755 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38973
I20260620 03:02:57.090421 16755 heartbeater.cc:464] Registering TS with master...
I20260620 03:02:57.091183 16755 heartbeater.cc:511] Master 127.0.97.62:38973 requested a full tablet report, sending...
I20260620 03:02:57.092857 16438 ts_manager.cc:194] Registered new tserver with Master: f293f78c15be405fb7883fcc06657c94 (127.0.97.2:38031)
I20260620 03:02:57.094415 16438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41331
W20260620 03:02:57.406980 16758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:02:57.407272 16758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:02:57.407442 16758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:02:57.414780 16758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:02:57.414892 16758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:02:57.415004 16758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:02:57.424649 16758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38337
--tserver_master_addrs=127.0.97.62:38973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:02:57.425931 16758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:02:57.427186 16758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:02:57.440038 16764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:57.439985 16766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:02:57.440227 16763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:02:57.441114 16758 server_base.cc:1034] running on GCE node
I20260620 03:02:57.441839 16758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:02:57.442819 16758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:02:57.444051 16758 hybrid_clock.cc:648] HybridClock initialized: now 1781924577444005 us; error 39 us; skew 500 ppm
I20260620 03:02:57.444422 16758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:02:57 dist-test-slave-f1j0 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:02:57.447625 16758 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:02:57.449117 16758 webserver.cc:460] Webserver started at http://127.0.97.3:38337/ using document root <none> and password file <none>
I20260620 03:02:57.449924 16758 fs_manager.cc:359] Metadata directory not provided
I20260620 03:02:57.450083 16758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:02:57.455708 16758 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 03:02:57.458441 16773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:57.459703 16758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:02:57.459887 16758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "5250fa83d69e4193bdc59611b502fecd"
format_stamp: "Formatted at 2026-06-20 03:02:34 on dist-test-slave-f1j0"
server_key: "efb12b833224ba46f228b1373eaf074e"
server_key_iv: "4985abc43ec99407ff6d724c515685e2"
server_key_version: "encryptionkey@0"
I20260620 03:02:57.460896 16758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:02:57.480067 16758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:02:57.484324 16758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:02:57.484628 16758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:02:57.485663 16758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:02:57.487020 16758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:02:57.487144 16758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:57.487286 16758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:02:57.487377 16758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:02:57.522644 16758 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37821
I20260620 03:02:57.522737 16885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37821 every 8 connection(s)
I20260620 03:02:57.524189 16758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:02:57.525224 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 16758
Jun 20 03:02:57 dist-test-slave-f1j0 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:02:57.541112 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:57.526441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33128 (local address 127.0.97.62:38973)
0620 03:02:57.526684 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:57.526693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:57.527102 (+ 409us) server_negotiation.cc:407] Connection header received
0620 03:02:57.527340 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:57.527345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:57.527441 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:57.527548 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:57.529759 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.530526 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.531914 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.532359 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.532494 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:57.535468 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:57.535474 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:57.535481 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:57.535518 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:57.538213 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:57.538922 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:57.538935 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:57.538937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:57.539056 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:57.539460 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:57.539467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:57.539471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:57.539824 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:57.540082 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:57.540415 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 03:02:57.540685 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 03:02:57.545230 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:57.527288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51275 (local address 127.0.97.62:38973)
0620 03:02:57.527782 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:57.527786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:57.528924 (+ 1138us) server_negotiation.cc:407] Connection header received
0620 03:02:57.530834 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:57.530839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:57.530915 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:57.531041 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:57.532516 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.533440 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.535029 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.535443 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.535579 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:57.539232 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:57.539237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:57.539240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:57.539317 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:57.542378 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:57.543188 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:57.543193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:57.543196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:57.543322 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:57.543832 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:57.543836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:57.543838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:57.544064 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:57.544168 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:57.544807 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 03:02:57.544964 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":278,"threads_started":1}
I20260620 03:02:57.546245 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:57.527995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:51275)
0620 03:02:57.528776 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:57.528820 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:57.530578 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:57.531286 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:57.531295 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:57.531784 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:02:57.532344 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:57.532360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.533654 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.533658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:57.534883 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:57.534895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.535666 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.535673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:57.535941 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:57.536607 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:02:57.536629 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:02:57.539010 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:02:57.542545 (+ 3535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:57.542554 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:57.542573 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:57.543059 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:57.543426 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:02:57.543429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:02:57.543431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:02:57.543690 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:02:57.544173 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:02:57.544179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:02:57.544498 (+ 319us) client_negotiation.cc:769] Sending connection context
0620 03:02:57.544717 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 03:02:57.545081 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":344,"threads_started":1}
I20260620 03:02:57.547569 16886 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38973
I20260620 03:02:57.547843 16886 heartbeater.cc:464] Registering TS with master...
I20260620 03:02:57.548539 16886 heartbeater.cc:511] Master 127.0.97.62:38973 requested a full tablet report, sending...
I20260620 03:02:57.549865 16438 ts_manager.cc:194] Registered new tserver with Master: 5250fa83d69e4193bdc59611b502fecd (127.0.97.3:37821)
I20260620 03:02:57.550740 16438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51275
I20260620 03:02:57.555986 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:02:57.562681 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:57.557032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33130 (local address 127.0.97.62:38973)
0620 03:02:57.557247 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:57.557253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:57.557636 (+ 383us) server_negotiation.cc:407] Connection header received
0620 03:02:57.557890 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:57.557894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:57.558023 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:57.558165 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:02:57.558603 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.559339 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.560796 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.561198 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.561328 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:57.561860 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:02:57.562056 (+ 196us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 03:02:57.562417 (+ 361us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33130: token expired
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 03:02:57.562822 16889 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33130: token expired
W20260620 03:02:57.563225 16359 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:38973)
I20260620 03:02:57.575543 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:57.563697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33144 (local address 127.0.97.62:38973)
0620 03:02:57.563905 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:57.563910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:57.563927 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:02:57.564083 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:57.564088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:57.564154 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:57.564253 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:57.565882 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.566555 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.567830 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.568291 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.568422 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:57.570901 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:57.570905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:57.570909 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:57.570935 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:57.573231 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:57.573887 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:57.573891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:57.573894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:57.574002 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:57.574322 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:57.574326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:57.574328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:57.574601 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:57.574801 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:57.575065 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 03:02:57.575215 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 03:02:57.578958 16438 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:02:57.586453 16438 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:02:57.620182 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:57.599371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41098 (local address 127.0.97.3:37821)
0620 03:02:57.600336 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:57.600349 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:57.601227 (+ 878us) server_negotiation.cc:407] Connection header received
0620 03:02:57.601361 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:57.601372 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:57.601729 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:57.601961 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:57.603213 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.613606 (+ 10393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.616710 (+ 3104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.618058 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.618220 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:57.618473 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:57.619613 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0620 03:02:57.619782 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":534,"threads_started":1}
I20260620 03:02:57.620195 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:57.599103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37765 (local address 127.0.0.1:56184)
0620 03:02:57.611163 (+ 12060us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:57.611181 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:57.611313 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:57.612104 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:57.612111 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:57.612137 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:57.612471 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:57.612480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.614705 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.614711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:57.617064 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:57.617079 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.619052 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.619058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:57.619347 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:57.619379 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 03:02:57.619469 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 03:02:57.619739 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11937,"mutex_wait_us":55,"thread_start_us":1465,"threads_started":1}
I20260620 03:02:57.620905 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:57.598623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37821 (local address 127.0.0.1:41098)
0620 03:02:57.600887 (+ 2264us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:57.600910 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:57.601064 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:57.602360 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:57.602371 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:57.602413 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:57.602936 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:57.602953 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.613973 (+ 11020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.613979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:57.616522 (+ 2543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:57.616535 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.618733 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.618744 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:57.619249 (+ 505us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:57.619323 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 03:02:57.619564 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 03:02:57.620531 (+ 967us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2105,"mutex_wait_us":1148,"thread_start_us":12261,"threads_started":1}
I20260620 03:02:57.620258 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:57.599737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56184 (local address 127.0.97.1:37765)
0620 03:02:57.610485 (+ 10748us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:57.610499 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:57.611449 (+ 950us) server_negotiation.cc:407] Connection header received
0620 03:02:57.611557 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:57.611565 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:57.611814 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:57.612022 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:57.612623 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.614493 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.617240 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.618680 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.618815 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:57.619038 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:57.619721 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 03:02:57.619907 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10490,"thread_start_us":10761,"threads_started":1}
I20260620 03:02:57.622670 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:57.598535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38031 (local address 127.0.0.1:51496)
0620 03:02:57.599341 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:57.599380 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:57.599542 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:57.613684 (+ 14142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:57.613690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:57.613719 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:57.614639 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:57.614658 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.617736 (+ 3078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.617740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:57.619736 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:57.619749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.621845 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.621852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:57.622031 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:57.622067 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 03:02:57.622160 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 03:02:57.622292 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":51,"thread_start_us":577,"threads_started":1}
I20260620 03:02:57.622720 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:57.598952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51496 (local address 127.0.97.2:38031)
0620 03:02:57.613062 (+ 14110us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:57.613071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:57.613097 (+ 26us) server_negotiation.cc:407] Connection header received
0620 03:02:57.613194 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:57.613201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:57.613429 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:57.615957 (+ 2528us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:57.616185 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.617539 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.619909 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.621604 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.621734 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:57.621927 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:57.622241 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 03:02:57.622415 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13953,"thread_start_us":13806,"threads_started":1}
I20260620 03:02:57.625243 16558 tablet_service.cc:1467] Processing CreateTablet for tablet 00c56355393e47e580eb6d074befe605 (DEFAULT_TABLE table=test-table [id=83e2d30b0d1c441c83f3d5414f56b1c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:02:57.625245 16821 tablet_service.cc:1467] Processing CreateTablet for tablet 00c56355393e47e580eb6d074befe605 (DEFAULT_TABLE table=test-table [id=83e2d30b0d1c441c83f3d5414f56b1c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:02:57.625880 16690 tablet_service.cc:1467] Processing CreateTablet for tablet 00c56355393e47e580eb6d074befe605 (DEFAULT_TABLE table=test-table [id=83e2d30b0d1c441c83f3d5414f56b1c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:02:57.626833 16821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c56355393e47e580eb6d074befe605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:02:57.626850 16558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c56355393e47e580eb6d074befe605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:02:57.627224 16690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c56355393e47e580eb6d074befe605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:02:57.640080 16902 tablet_bootstrap.cc:492] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94: Bootstrap starting.
I20260620 03:02:57.642412 16903 tablet_bootstrap.cc:492] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162: Bootstrap starting.
I20260620 03:02:57.644042 16902 tablet_bootstrap.cc:654] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94: Neither blocks nor log segments found. Creating new log.
I20260620 03:02:57.644191 16904 tablet_bootstrap.cc:492] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd: Bootstrap starting.
I20260620 03:02:57.645812 16623 heartbeater.cc:503] Master 127.0.97.62:38973 was elected leader, sending a full tablet report...
I20260620 03:02:57.645998 16903 tablet_bootstrap.cc:654] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162: Neither blocks nor log segments found. Creating new log.
I20260620 03:02:57.647286 16904 tablet_bootstrap.cc:654] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd: Neither blocks nor log segments found. Creating new log.
I20260620 03:02:57.647305 16903 log.cc:824] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:57.647395 16902 log.cc:824] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:57.648481 16904 log.cc:824] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd: Log is configured to *not* fsync() on all Append() calls
I20260620 03:02:57.653921 16903 tablet_bootstrap.cc:492] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162: No bootstrap required, opened a new log
I20260620 03:02:57.653927 16904 tablet_bootstrap.cc:492] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd: No bootstrap required, opened a new log
I20260620 03:02:57.654222 16904 ts_tablet_manager.cc:1397] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260620 03:02:57.654244 16903 ts_tablet_manager.cc:1397] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260620 03:02:57.654608 16902 tablet_bootstrap.cc:492] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94: No bootstrap required, opened a new log
I20260620 03:02:57.654867 16902 ts_tablet_manager.cc:1397] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.007s
I20260620 03:02:57.661224 16903 raft_consensus.cc:348] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } }
I20260620 03:02:57.661623 16903 raft_consensus.cc:374] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:02:57.661731 16903 raft_consensus.cc:729] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39a5cebd2c947848cd4f262dac7d162, State: Initialized, Role: FOLLOWER
I20260620 03:02:57.662353 16903 consensus_queue.cc:260] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } }
I20260620 03:02:57.664211 16903 ts_tablet_manager.cc:1428] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260620 03:02:57.664911 16904 raft_consensus.cc:348] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } }
I20260620 03:02:57.665366 16904 raft_consensus.cc:374] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:02:57.665472 16904 raft_consensus.cc:729] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5250fa83d69e4193bdc59611b502fecd, State: Initialized, Role: FOLLOWER
I20260620 03:02:57.665362 16902 raft_consensus.cc:348] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } }
I20260620 03:02:57.665781 16902 raft_consensus.cc:374] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:02:57.665869 16902 raft_consensus.cc:729] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f293f78c15be405fb7883fcc06657c94, State: Initialized, Role: FOLLOWER
I20260620 03:02:57.666096 16904 consensus_queue.cc:260] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } }
I20260620 03:02:57.666404 16902 consensus_queue.cc:260] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } }
I20260620 03:02:57.667649 16755 heartbeater.cc:503] Master 127.0.97.62:38973 was elected leader, sending a full tablet report...
I20260620 03:02:57.668329 16886 heartbeater.cc:503] Master 127.0.97.62:38973 was elected leader, sending a full tablet report...
I20260620 03:02:57.668977 16904 ts_tablet_manager.cc:1428] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260620 03:02:57.669799 16902 ts_tablet_manager.cc:1428] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260620 03:02:57.776144 16887 tablet.cc:2369] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:02:57.781740 16909 raft_consensus.cc:482] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:02:57.782050 16909 raft_consensus.cc:504] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } }
I20260620 03:02:57.783874 16909 leader_election.cc:288] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f293f78c15be405fb7883fcc06657c94 (127.0.97.2:38031), a39a5cebd2c947848cd4f262dac7d162 (127.0.97.1:37765)
I20260620 03:02:57.790437 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:57.784367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33865 (local address 127.0.97.1:37765)
0620 03:02:57.784545 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:57.784552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:57.784650 (+ 98us) server_negotiation.cc:407] Connection header received
0620 03:02:57.784894 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:57.784903 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:57.784983 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:57.785098 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:57.785539 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.786599 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.788430 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.789472 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.789636 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:57.789723 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:57.790060 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 03:02:57.790157 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 03:02:57.790462 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:57.784191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37765 (local address 127.0.97.3:33865)
0620 03:02:57.784513 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:57.784539 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:57.784718 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:57.785142 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:57.785150 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:57.785184 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:57.785394 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:57.785405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.786725 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.786730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:57.788237 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:57.788249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.789672 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.789678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:57.789872 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:57.789920 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 03:02:57.790022 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 03:02:57.790160 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260620 03:02:57.791201 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:57.784318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39159 (local address 127.0.97.2:38031)
0620 03:02:57.784531 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:57.784538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:57.785178 (+ 640us) server_negotiation.cc:407] Connection header received
0620 03:02:57.785452 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:57.785459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:57.785550 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:57.785673 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:02:57.786300 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.787217 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.789105 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.790133 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.790257 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:57.790356 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:57.790815 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 03:02:57.790957 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 03:02:57.791760 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:57.784235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38031 (local address 127.0.97.3:39159)
0620 03:02:57.785061 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 03:02:57.785081 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:02:57.785284 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:02:57.785707 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:02:57.785714 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:02:57.785741 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:02:57.786091 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:57.786104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.787367 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.787370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:57.788946 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:02:57.788960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.790368 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.790374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:02:57.790577 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:57.790619 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 03:02:57.791383 (+ 764us) client_negotiation.cc:240] Negotiation successful
0620 03:02:57.791490 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"spinlock_wait_cycles":47488,"thread_start_us":480,"threads_started":1}
I20260620 03:02:57.791627 16578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c56355393e47e580eb6d074befe605" candidate_uuid: "5250fa83d69e4193bdc59611b502fecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39a5cebd2c947848cd4f262dac7d162" is_pre_election: true
I20260620 03:02:57.792188 16578 raft_consensus.cc:2454] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5250fa83d69e4193bdc59611b502fecd in term 0.
I20260620 03:02:57.792830 16710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c56355393e47e580eb6d074befe605" candidate_uuid: "5250fa83d69e4193bdc59611b502fecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f293f78c15be405fb7883fcc06657c94" is_pre_election: true
I20260620 03:02:57.792972 16775 leader_election.cc:302] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5250fa83d69e4193bdc59611b502fecd, a39a5cebd2c947848cd4f262dac7d162; no voters:
I20260620 03:02:57.793392 16909 raft_consensus.cc:2781] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:02:57.793351 16710 raft_consensus.cc:2454] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5250fa83d69e4193bdc59611b502fecd in term 0.
I20260620 03:02:57.793548 16909 raft_consensus.cc:482] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:02:57.793676 16909 raft_consensus.cc:3037] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:02:57.795997 16909 raft_consensus.cc:504] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } }
I20260620 03:02:57.796528 16909 leader_election.cc:288] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [CANDIDATE]: Term 1 election: Requested vote from peers f293f78c15be405fb7883fcc06657c94 (127.0.97.2:38031), a39a5cebd2c947848cd4f262dac7d162 (127.0.97.1:37765)
I20260620 03:02:57.797034 16710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c56355393e47e580eb6d074befe605" candidate_uuid: "5250fa83d69e4193bdc59611b502fecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f293f78c15be405fb7883fcc06657c94"
I20260620 03:02:57.797065 16578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c56355393e47e580eb6d074befe605" candidate_uuid: "5250fa83d69e4193bdc59611b502fecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39a5cebd2c947848cd4f262dac7d162"
I20260620 03:02:57.797250 16578 raft_consensus.cc:3037] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:02:57.797286 16710 raft_consensus.cc:3037] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:02:57.799461 16578 raft_consensus.cc:2454] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5250fa83d69e4193bdc59611b502fecd in term 1.
I20260620 03:02:57.799608 16710 raft_consensus.cc:2454] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5250fa83d69e4193bdc59611b502fecd in term 1.
I20260620 03:02:57.799890 16775 leader_election.cc:302] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5250fa83d69e4193bdc59611b502fecd, a39a5cebd2c947848cd4f262dac7d162; no voters:
I20260620 03:02:57.800282 16909 raft_consensus.cc:2781] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:02:57.800733 16909 raft_consensus.cc:686] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [term 1 LEADER]: Becoming Leader. State: Replica: 5250fa83d69e4193bdc59611b502fecd, State: Running, Role: LEADER
I20260620 03:02:57.801213 16909 consensus_queue.cc:237] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } }
I20260620 03:02:57.805754 16438 catalog_manager.cc:5495] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd reported cstate change: term changed from 0 to 1, leader changed from <none> to 5250fa83d69e4193bdc59611b502fecd (127.0.97.3). New cstate: current_term: 1 leader_uuid: "5250fa83d69e4193bdc59611b502fecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5250fa83d69e4193bdc59611b502fecd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 } health_report { overall_health: UNKNOWN } } }
W20260620 03:02:57.818676 16756 tablet.cc:2369] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:02:57.859608 16624 tablet.cc:2369] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:02:57.863842 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:57.858516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.0.97.3:37821)
0620 03:02:57.858741 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:57.858746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:57.858764 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:02:57.858938 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:57.858942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:57.859025 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:57.859147 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:02:57.859534 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.860297 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.861697 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.862104 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.862245 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:57.862753 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:02:57.863244 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:02:57.863414 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:57.863517 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 03:02:57.863598 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 03:02:57.871803 16578 raft_consensus.cc:1261] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162 [term 1 FOLLOWER]: Refusing update from remote peer 5250fa83d69e4193bdc59611b502fecd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:02:57.871805 16710 raft_consensus.cc:1261] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94 [term 1 FOLLOWER]: Refusing update from remote peer 5250fa83d69e4193bdc59611b502fecd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:02:57.872728 16912 consensus_queue.cc:1035] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a39a5cebd2c947848cd4f262dac7d162" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:02:57.873064 16909 consensus_queue.cc:1035] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f293f78c15be405fb7883fcc06657c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:02:57.884356 16915 mvcc.cc:205] Tried to move back new op lower bound from 7298763070953426944 to 7298763070677590016. Current Snapshot: MvccSnapshot[applied={T|T < 7298763070953426944}]
I20260620 03:02:57.899760 16438 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:02:57.900076 16438 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:02:57.903806 16438 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19: Sending DeleteTablet for 3 replicas of tablet 00c56355393e47e580eb6d074befe605
I20260620 03:02:57.905076 16821 tablet_service.cc:1514] Processing DeleteTablet for tablet 00c56355393e47e580eb6d074befe605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:02:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41098
I20260620 03:02:57.905035 16690 tablet_service.cc:1514] Processing DeleteTablet for tablet 00c56355393e47e580eb6d074befe605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:02:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51496
I20260620 03:02:57.905321 16558 tablet_service.cc:1514] Processing DeleteTablet for tablet 00c56355393e47e580eb6d074befe605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:02:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56184
I20260620 03:02:57.906463 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 16475
I20260620 03:02:57.919533 16926 tablet_replica.cc:330] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162: stopping tablet replica
I20260620 03:02:57.920147 16926 raft_consensus.cc:2229] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:02:57.920733 16926 raft_consensus.cc:2258] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:02:57.922391 16925 tablet_replica.cc:330] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd: stopping tablet replica
I20260620 03:02:57.922549 16924 tablet_replica.cc:330] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94: stopping tablet replica
I20260620 03:02:57.922976 16924 raft_consensus.cc:2229] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:02:57.923111 16926 ts_tablet_manager.cc:1905] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:02:57.923478 16925 raft_consensus.cc:2229] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:02:57.924010 16924 raft_consensus.cc:2258] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:02:57.924674 16925 raft_consensus.cc:2258] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:02:57.927238 16925 ts_tablet_manager.cc:1905] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:02:57.927807 16924 ts_tablet_manager.cc:1905] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:02:57.930214 16926 ts_tablet_manager.cc:1918] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:02:57.930371 16926 log.cc:1196] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/00c56355393e47e580eb6d074befe605
I20260620 03:02:57.930917 16926 ts_tablet_manager.cc:1939] T 00c56355393e47e580eb6d074befe605 P a39a5cebd2c947848cd4f262dac7d162: Deleting consensus metadata
I20260620 03:02:57.932873 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:57.907854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56200 (local address 127.0.97.1:37765)
0620 03:02:57.908134 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:57.908143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:57.908169 (+ 26us) server_negotiation.cc:407] Connection header received
0620 03:02:57.908244 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:57.908252 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:57.908381 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:57.908523 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:57.911035 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.912629 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.915245 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:57.916181 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:57.916329 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:57.920630 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:57.920647 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:57.920665 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:57.920713 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:57.924822 (+ 4109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:57.926289 (+ 1467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:57.926298 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:57.926302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:57.926474 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:57.929986 (+ 3512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:57.929992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:57.929995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:57.930531 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:57.930882 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:57.932250 (+ 1368us) server_negotiation.cc:299] Negotiation successful
0620 03:02:57.932493 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 03:02:57.932909 16423 catalog_manager.cc:4841] TS a39a5cebd2c947848cd4f262dac7d162 (127.0.97.1:37765): tablet 00c56355393e47e580eb6d074befe605 (table test-table [id=83e2d30b0d1c441c83f3d5414f56b1c0]) successfully deleted
I20260620 03:02:57.933597 16618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:57.935353 16925 ts_tablet_manager.cc:1918] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:02:57.935614 16925 log.cc:1196] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/00c56355393e47e580eb6d074befe605
I20260620 03:02:57.936321 16925 ts_tablet_manager.cc:1939] T 00c56355393e47e580eb6d074befe605 P 5250fa83d69e4193bdc59611b502fecd: Deleting consensus metadata
I20260620 03:02:57.938611 16425 catalog_manager.cc:4841] TS 5250fa83d69e4193bdc59611b502fecd (127.0.97.3:37821): tablet 00c56355393e47e580eb6d074befe605 (table test-table [id=83e2d30b0d1c441c83f3d5414f56b1c0]) successfully deleted
I20260620 03:02:57.939119 16924 ts_tablet_manager.cc:1918] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:02:57.939277 16924 log.cc:1196] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/00c56355393e47e580eb6d074befe605
I20260620 03:02:57.939831 16924 ts_tablet_manager.cc:1939] T 00c56355393e47e580eb6d074befe605 P f293f78c15be405fb7883fcc06657c94: Deleting consensus metadata
I20260620 03:02:57.942411 16422 catalog_manager.cc:4841] TS f293f78c15be405fb7883fcc06657c94 (127.0.97.2:38031): tablet 00c56355393e47e580eb6d074befe605 (table test-table [id=83e2d30b0d1c441c83f3d5414f56b1c0]) successfully deleted
I20260620 03:02:58.045845 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 16475
I20260620 03:02:58.063673 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 16627
I20260620 03:02:58.078881 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:58.065334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.0.97.2:38031)
0620 03:02:58.065526 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:58.065530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:58.065545 (+ 15us) server_negotiation.cc:407] Connection header received
0620 03:02:58.065636 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:58.065641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:58.065714 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:58.065865 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:58.067189 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:58.068094 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:58.069353 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:58.069977 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:58.070111 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:58.072776 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:58.072788 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:58.072812 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:58.072853 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:58.075922 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:58.076609 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:58.076618 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:58.076622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:58.076758 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:58.077145 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:58.077151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:58.077154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:58.077543 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:58.077852 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:58.078201 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 03:02:58.078430 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 03:02:58.079619 16750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:58.191144 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 16627
I20260620 03:02:58.209514 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 16758
I20260620 03:02:58.226281 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:02:58.210407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.0.97.3:37821)
0620 03:02:58.210581 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 03:02:58.210585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:02:58.210602 (+ 17us) server_negotiation.cc:407] Connection header received
0620 03:02:58.210766 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:02:58.210770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:02:58.210856 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:02:58.210938 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:02:58.212822 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:58.213714 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:58.215587 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:02:58.216068 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:02:58.216165 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:02:58.219544 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:02:58.219555 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:02:58.219570 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:02:58.219611 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:02:58.223383 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:58.224186 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:58.224193 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:58.224195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:58.224332 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:02:58.224749 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:02:58.224752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:02:58.224754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:02:58.225090 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:02:58.225337 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 03:02:58.225743 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 03:02:58.225917 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 03:02:58.227069 16881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:58.357141 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 16758
I20260620 03:02:58.379626 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 16405
I20260620 03:02:58.380795 16468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:02:58.494053 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 16405
I20260620 03:03:18.516942 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38973
--webserver_interface=127.0.97.62
--webserver_port=44719
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:03:18.848160 16931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:18.848436 16931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:18.848639 16931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:18.855759 16931 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 03:03:18.855854 16931 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 03:03:18.855937 16931 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 03:03:18.856031 16931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:03:18.856117 16931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:18.856216 16931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:18.856281 16931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:03:18.856364 16931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:03:18.865926 16931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:18.867206 16931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:18.868530 16931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:18.880470 16937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:18.880494 16939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:18.881117 16936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:18.881214 16931 server_base.cc:1034] running on GCE node
I20260620 03:03:18.882128 16931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:18.883544 16931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:18.884785 16931 hybrid_clock.cc:648] HybridClock initialized: now 1781924598884716 us; error 34 us; skew 500 ppm
I20260620 03:03:18.885154 16931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:03:18 dist-test-slave-f1j0 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:03:18.893216 16931 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:03:18.895071 16931 webserver.cc:460] Webserver started at http://127.0.97.62:44719/ using document root <none> and password file <none>
I20260620 03:03:18.895869 16931 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:18.895995 16931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:18.901918 16931 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 03:03:18.905026 16946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:18.906562 16931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:03:18.906759 16931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "b717ef0d53a04f0da438426a02ac3c19"
format_stamp: "Formatted at 2026-06-20 03:02:32 on dist-test-slave-f1j0"
server_key: "e86a41db6fb6925d8d778875532d1db2"
server_key_iv: "702d2e69bb19e84e9ded96ce919dc5de"
server_key_version: "encryptionkey@0"
I20260620 03:03:18.907743 16931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:18.924978 16931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:18.929131 16931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:18.929463 16931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:18.949321 16931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38973
I20260620 03:03:18.949343 16997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38973 every 8 connection(s)
I20260620 03:03:18.950946 16931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:03:18.958523 16998 sys_catalog.cc:263] Verifying existing consensus state
I20260620 03:03:18.960530 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 16931
I20260620 03:03:18.961615 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:37765
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38973
--webserver_port=38099
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:03:18.963277 16998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19: Bootstrap starting.
I20260620 03:03:18.987867 16998 log.cc:824] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:19.005528 16998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 03:03:19.006201 16998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19: Bootstrap complete.
I20260620 03:03:19.016759 16998 raft_consensus.cc:348] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b717ef0d53a04f0da438426a02ac3c19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } }
I20260620 03:03:19.017855 16998 raft_consensus.cc:729] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b717ef0d53a04f0da438426a02ac3c19, State: Initialized, Role: FOLLOWER
I20260620 03:03:19.018538 16998 consensus_queue.cc:260] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b717ef0d53a04f0da438426a02ac3c19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } }
I20260620 03:03:19.018795 16998 raft_consensus.cc:388] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:03:19.018925 16998 raft_consensus.cc:482] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:03:19.019119 16998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 2 FOLLOWER]: Advancing to term 3
I20260620 03:03:19.023397 16998 raft_consensus.cc:504] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b717ef0d53a04f0da438426a02ac3c19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } }
I20260620 03:03:19.023928 16998 leader_election.cc:302] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b717ef0d53a04f0da438426a02ac3c19; no voters:
I20260620 03:03:19.024659 16998 leader_election.cc:288] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 03:03:19.024825 17002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 03:03:19.026041 17002 raft_consensus.cc:686] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [term 3 LEADER]: Becoming Leader. State: Replica: b717ef0d53a04f0da438426a02ac3c19, State: Running, Role: LEADER
I20260620 03:03:19.026736 17002 consensus_queue.cc:237] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b717ef0d53a04f0da438426a02ac3c19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } }
I20260620 03:03:19.027107 16998 sys_catalog.cc:564] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:03:19.029469 17004 sys_catalog.cc:455] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b717ef0d53a04f0da438426a02ac3c19. Latest consensus state: current_term: 3 leader_uuid: "b717ef0d53a04f0da438426a02ac3c19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b717ef0d53a04f0da438426a02ac3c19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } } }
I20260620 03:03:19.029870 17004 sys_catalog.cc:458] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [sys.catalog]: This master's current role is: LEADER
I20260620 03:03:19.029626 17003 sys_catalog.cc:455] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b717ef0d53a04f0da438426a02ac3c19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b717ef0d53a04f0da438426a02ac3c19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } } }
I20260620 03:03:19.030079 17003 sys_catalog.cc:458] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19 [sys.catalog]: This master's current role is: LEADER
I20260620 03:03:19.039351 17009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:03:19.051599 17009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:03:19.054051 17009 catalog_manager.cc:1259] Loaded cluster ID: fde1555b54fe4d80a7f199e742724d7f
I20260620 03:03:19.054262 17009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:03:19.060158 17009 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:03:19.062888 17009 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19: Loaded TSK: 0
I20260620 03:03:19.063185 17009 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b717ef0d53a04f0da438426a02ac3c19: Loaded TSK: 1
F20260620 03:03:19.064666 17009 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2cd51d79ad google::LogMessage::Fail() at ??:0
@ 0x7f2cd51da88a google::LogMessage::SendToLog() at ??:0
@ 0x7f2cd51d74de google::LogMessage::Flush() at ??:0
@ 0x7f2cd51d8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2cd8f3b7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f2cd8f2a73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f2ceac061c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f2ceac05783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f2ceac912a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f2ceac910c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f2ceacdb5e9 std::function<>::operator()() at ??:0
@ 0x7f2ceac09cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f2ceac07c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2ceac8f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f2ceac8f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2cd7e23345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2cd7e2691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f2cd7e26799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2cd7e01eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2cdb0db6db start_thread at ??:0
@ 0x7f2cd407161f clone at ??:0
W20260620 03:03:19.383536 17000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:19.383821 17000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:19.383915 17000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:19.391265 17000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:19.391381 17000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:19.391495 17000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:03:19.401111 17000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:37765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38099
--tserver_master_addrs=127.0.97.62:38973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:19.402433 17000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:19.403685 17000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:19.416539 17045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:19.416584 17046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:19.416615 17048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:19.417876 17000 server_base.cc:1034] running on GCE node
I20260620 03:03:19.418521 17000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:19.419451 17000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:19.420684 17000 hybrid_clock.cc:648] HybridClock initialized: now 1781924599420630 us; error 37 us; skew 500 ppm
I20260620 03:03:19.421046 17000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:03:19 dist-test-slave-f1j0 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:03:19.424418 17000 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:03:19.426152 17000 webserver.cc:460] Webserver started at http://127.0.97.1:38099/ using document root <none> and password file <none>
I20260620 03:03:19.426893 17000 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:19.427029 17000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:19.432397 17000 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 03:03:19.435192 17055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:19.436534 17000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 03:03:19.436729 17000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "a39a5cebd2c947848cd4f262dac7d162"
format_stamp: "Formatted at 2026-06-20 03:02:33 on dist-test-slave-f1j0"
server_key: "db7fe6df93ebe21ae117a4d74f29e4cb"
server_key_iv: "e63a6b7b700a27b57a82ff8d23cf8d5e"
server_key_version: "encryptionkey@0"
I20260620 03:03:19.437852 17000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:19.452545 17000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:19.456914 17000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:19.457194 17000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:19.458185 17000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:19.459504 17000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:19.459650 17000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:19.459772 17000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:19.459867 17000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:19.493394 17000 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37765
I20260620 03:03:19.493479 17167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37765 every 8 connection(s)
I20260620 03:03:19.494899 17000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:03:19.499756 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:19.497973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:60307)
0620 03:03:19.498693 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:19.498883 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":270,"threads_started":1}
I20260620 03:03:19.500025 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 17000
W20260620 03:03:19.500501 17168 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38973 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38973: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
I20260620 03:03:19.500830 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38031
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38973
--webserver_port=42335
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:03:19.501390 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:19.501087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:45823)
0620 03:03:19.501222 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:19.501288 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 03:03:19.502259 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:19.501902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:34507)
0620 03:03:19.502036 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:19.502118 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 03:03:19.502512 17168 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 03:03:19.826000 17171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:19.826300 17171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:19.826408 17171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:19.833667 17171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:19.833771 17171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:19.833882 17171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:03:19.843917 17171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42335
--tserver_master_addrs=127.0.97.62:38973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:19.845104 17171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:19.846360 17171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:19.858808 17176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:19.858880 17177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:19.858904 17179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:19.859719 17171 server_base.cc:1034] running on GCE node
I20260620 03:03:19.860304 17171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:19.861197 17171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:19.862412 17171 hybrid_clock.cc:648] HybridClock initialized: now 1781924599862355 us; error 46 us; skew 500 ppm
I20260620 03:03:19.862727 17171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:03:19 dist-test-slave-f1j0 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:03:19.865808 17171 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:03:19.867188 17171 webserver.cc:460] Webserver started at http://127.0.97.2:42335/ using document root <none> and password file <none>
I20260620 03:03:19.867808 17171 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:19.867908 17171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:19.873170 17171 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:03:19.875821 17186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:19.877193 17171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 03:03:19.877351 17171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "f293f78c15be405fb7883fcc06657c94"
format_stamp: "Formatted at 2026-06-20 03:02:34 on dist-test-slave-f1j0"
server_key: "c7f2e5617c9cfc4cf25da8c2f55a96ee"
server_key_iv: "d35d1aaddd17e39c6712045f290b70bd"
server_key_version: "encryptionkey@0"
I20260620 03:03:19.878429 17171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:19.890905 17171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:19.895455 17171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:19.895745 17171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:19.896796 17171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:19.898234 17171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:19.898320 17171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:19.898485 17171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:19.898569 17171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:19.931751 17171 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38031
I20260620 03:03:19.931862 17298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38031 every 8 connection(s)
I20260620 03:03:19.933197 17171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:03:19.933874 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 17171
I20260620 03:03:19.934592 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37821
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38973
--webserver_port=38337
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:03:19.938838 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:19.936531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:51305)
0620 03:03:19.937385 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:19.937676 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":320,"threads_started":1}
W20260620 03:03:19.939831 17299 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38973 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38973: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
I20260620 03:03:19.942449 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:19.940450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:36195)
0620 03:03:19.942234 (+ 1784us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:19.942315 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1696,"threads_started":1}
I20260620 03:03:19.944058 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:19.943428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:56245)
0620 03:03:19.943676 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:19.943817 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260620 03:03:19.944370 17299 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 03:03:20.271842 17303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:20.272123 17303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:20.272281 17303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:20.279783 17303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:20.279886 17303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:20.280086 17303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:03:20.290081 17303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38337
--tserver_master_addrs=127.0.97.62:38973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:20.291356 17303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:20.292627 17303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:20.305665 17308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:20.305717 17309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:20.305619 17311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:20.306248 17303 server_base.cc:1034] running on GCE node
I20260620 03:03:20.306852 17303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:20.307739 17303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:20.308966 17303 hybrid_clock.cc:648] HybridClock initialized: now 1781924600308912 us; error 45 us; skew 500 ppm
I20260620 03:03:20.309283 17303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:03:20 dist-test-slave-f1j0 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:03:20.312481 17303 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:03:20.313947 17303 webserver.cc:460] Webserver started at http://127.0.97.3:38337/ using document root <none> and password file <none>
I20260620 03:03:20.314563 17303 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:20.314661 17303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:20.319759 17303 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 03:03:20.322480 17318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:20.323882 17303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 03:03:20.324021 17303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "5250fa83d69e4193bdc59611b502fecd"
format_stamp: "Formatted at 2026-06-20 03:02:34 on dist-test-slave-f1j0"
server_key: "efb12b833224ba46f228b1373eaf074e"
server_key_iv: "4985abc43ec99407ff6d724c515685e2"
server_key_version: "encryptionkey@0"
I20260620 03:03:20.324940 17303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:20.344630 17303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:20.349013 17303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:20.349234 17303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:20.350212 17303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:20.351570 17303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:20.351655 17303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:20.351794 17303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:20.351913 17303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:20.386600 17303 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37821
I20260620 03:03:20.386698 17430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37821 every 8 connection(s)
I20260620 03:03:20.388103 17303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:03:20.391314 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 17303
I20260620 03:03:20.393522 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:20.391393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:44125)
0620 03:03:20.392242 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:20.392471 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":341,"threads_started":1}
W20260620 03:03:20.394202 17431 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38973 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38973: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
I20260620 03:03:20.394950 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:20.394660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:35229)
0620 03:03:20.394784 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:20.394847 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 03:03:20.395694 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:20.395403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:57569)
0620 03:03:20.395524 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:20.395591 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 03:03:20.395922 17431 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 03:03:20.503933 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:20.503073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:55491)
0620 03:03:20.503563 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:20.503694 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":258,"threads_started":1}
I20260620 03:03:20.945783 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:20.944953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:47469)
0620 03:03:20.945395 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:20.945486 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":215,"threads_started":1}
I20260620 03:03:21.397193 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:21.396479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:42941)
0620 03:03:21.396896 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:21.396977 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":209,"threads_started":1}
I20260620 03:03:21.505859 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:21.504909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:55091)
0620 03:03:21.505470 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:21.505611 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":276,"threads_started":1}
I20260620 03:03:21.947345 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:21.946665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:36469)
0620 03:03:21.947100 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:21.947182 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":238,"threads_started":1}
I20260620 03:03:22.399014 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:22.398101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:60321)
0620 03:03:22.398652 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:22.398780 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":247,"threads_started":1}
I20260620 03:03:22.507449 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:22.506703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:39205)
0620 03:03:22.507160 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:22.507244 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":244,"threads_started":1}
I20260620 03:03:22.948964 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:22.948257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:60579)
0620 03:03:22.948710 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:22.948791 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":249,"threads_started":1}
I20260620 03:03:23.400673 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:23.399954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:46301)
0620 03:03:23.400426 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:23.400508 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":262,"threads_started":1}
I20260620 03:03:23.509099 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:23.508377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:54647)
0620 03:03:23.508820 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:23.508903 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":213,"threads_started":1}
I20260620 03:03:23.950757 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:23.949946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:37471)
0620 03:03:23.950477 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:23.950560 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":286,"threads_started":1}
I20260620 03:03:24.402562 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:24.401719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:45579)
0620 03:03:24.402288 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:24.402362 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":347,"threads_started":1}
I20260620 03:03:24.510831 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:24.510111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:47403)
0620 03:03:24.510580 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:24.510661 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":239,"threads_started":1}
I20260620 03:03:24.952538 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:24.951723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:37841)
0620 03:03:24.952245 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:24.952344 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":251,"threads_started":1}
I20260620 03:03:25.404174 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:25.403439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:44553)
0620 03:03:25.403929 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:25.404007 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":252,"threads_started":1}
I20260620 03:03:25.512691 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:25.511854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:50001)
0620 03:03:25.512355 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:25.512490 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":277,"threads_started":1}
I20260620 03:03:25.954412 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:25.953687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:56955)
0620 03:03:25.954128 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:25.954228 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":225,"threads_started":1}
I20260620 03:03:26.405876 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:26.405051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:34649)
0620 03:03:26.405605 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:26.405714 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":304,"threads_started":1}
I20260620 03:03:26.514370 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:26.513515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:45519)
0620 03:03:26.514075 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:26.514153 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":218,"threads_started":1}
I20260620 03:03:26.956272 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:26.955442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:37949)
0620 03:03:26.955948 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:26.956048 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":250,"threads_started":1}
I20260620 03:03:27.407562 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:27.406778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:51859)
0620 03:03:27.407260 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:27.407335 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":226,"threads_started":1}
I20260620 03:03:27.516039 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:27.515300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:49241)
0620 03:03:27.515773 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:27.515855 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":264,"threads_started":1}
I20260620 03:03:27.958040 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:27.957256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:46393)
0620 03:03:27.957782 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:27.957867 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":260,"threads_started":1}
I20260620 03:03:28.409327 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:28.408586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:34393)
0620 03:03:28.409071 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:28.409151 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":260,"threads_started":1}
I20260620 03:03:28.517913 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:28.517002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:57697)
0620 03:03:28.517525 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:28.517674 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":269,"threads_started":1}
I20260620 03:03:28.959571 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:28.958884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:39787)
0620 03:03:28.959306 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:28.959395 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":223,"threads_started":1}
I20260620 03:03:29.410951 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:29.410255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:56527)
0620 03:03:29.410706 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:29.410781 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":224,"threads_started":1}
I20260620 03:03:29.519505 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:29.518791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:41407)
0620 03:03:29.519251 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:29.519339 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":243,"threads_started":1}
I20260620 03:03:29.961215 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:29.960547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:49963)
0620 03:03:29.960981 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:29.961058 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":216,"threads_started":1}
I20260620 03:03:30.412786 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:30.411903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:35883)
0620 03:03:30.412508 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:30.412625 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":368,"threads_started":1}
I20260620 03:03:30.521170 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:30.520432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:39605)
0620 03:03:30.520904 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:30.520993 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":265,"threads_started":1}
I20260620 03:03:30.962952 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:30.962198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:53451)
0620 03:03:30.962677 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:30.962757 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":273,"threads_started":1}
I20260620 03:03:31.414552 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:31.413810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:38759)
0620 03:03:31.414247 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:31.414350 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":225,"threads_started":1}
I20260620 03:03:31.523016 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:31.522276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:54401)
0620 03:03:31.522764 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:31.522857 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":250,"threads_started":1}
I20260620 03:03:31.964838 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:31.964068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:48071)
0620 03:03:31.964578 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:31.964671 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":266,"threads_started":1}
I20260620 03:03:32.416368 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:32.415594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:34967)
0620 03:03:32.416071 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:32.416158 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":247,"threads_started":1}
I20260620 03:03:32.524801 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:32.524057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:57321)
0620 03:03:32.524525 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:32.524607 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":218,"threads_started":1}
I20260620 03:03:32.966569 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:32.965858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:38219)
0620 03:03:32.966312 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:32.966387 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":228,"threads_started":1}
I20260620 03:03:33.418200 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:33.417418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:44327)
0620 03:03:33.417922 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:33.418007 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":276,"threads_started":1}
I20260620 03:03:33.526605 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:33.525882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:45259)
0620 03:03:33.526312 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:33.526415 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":217,"threads_started":1}
I20260620 03:03:33.968241 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:33.967454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:54195)
0620 03:03:33.967960 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:33.968053 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":287,"threads_started":1}
I20260620 03:03:34.419988 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:34.419194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:48009)
0620 03:03:34.419708 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:34.419812 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":267,"threads_started":1}
I20260620 03:03:34.528549 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:34.527732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:35671)
0620 03:03:34.528206 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:34.528290 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":242,"threads_started":1}
I20260620 03:03:34.970191 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:34.969471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:56353)
0620 03:03:34.969932 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:34.970017 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":265,"threads_started":1}
I20260620 03:03:35.393437 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 17000
I20260620 03:03:35.415482 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:35.394637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40744 (local address 127.0.97.1:37765)
0620 03:03:35.395244 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:35.395268 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:35.395311 (+ 43us) server_negotiation.cc:407] Connection header received
0620 03:03:35.396918 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:35.396929 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:35.397377 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:35.397772 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:35.399088 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:35.400352 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:35.401676 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:35.402355 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:35.402504 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:35.409126 (+ 6622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:35.409135 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:35.409153 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:35.409196 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:35.412172 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:35.412909 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:35.412917 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:35.412924 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:35.413123 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:35.413488 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:35.413493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:35.413495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:35.414171 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:35.414439 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:35.414798 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 03:03:35.415162 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":312,"threads_started":1}
I20260620 03:03:35.416914 17163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:35.421980 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:35.421165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:59165)
0620 03:03:35.421710 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:35.421800 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":244,"threads_started":1}
I20260620 03:03:35.503403 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 17000
I20260620 03:03:35.520695 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 17171
I20260620 03:03:35.539386 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:35.521737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43210 (local address 127.0.97.2:38031)
0620 03:03:35.522286 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:35.522300 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:35.522336 (+ 36us) server_negotiation.cc:407] Connection header received
0620 03:03:35.523892 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:35.523901 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:35.524356 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:35.524732 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:35.526292 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:35.527535 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:35.528963 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:35.529866 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:35.530072 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:35.532790 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:35.532798 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:35.532814 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:35.532856 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:35.535917 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:35.536680 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:35.536688 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:35.536695 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:35.536870 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:35.537257 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:35.537264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:35.537267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:35.537873 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:35.538199 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:35.538681 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 03:03:35.539097 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":277,"threads_started":1}
I20260620 03:03:35.540774 17294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:35.628347 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 17171
I20260620 03:03:35.647285 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 17303
I20260620 03:03:35.665853 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:35.648340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54350 (local address 127.0.97.3:37821)
0620 03:03:35.648945 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:35.648960 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:35.649009 (+ 49us) server_negotiation.cc:407] Connection header received
0620 03:03:35.650797 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:35.650813 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:35.651271 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:35.651631 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:35.653077 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:35.654397 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:35.655652 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:35.656313 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:35.656506 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:35.659142 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:35.659150 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:35.659165 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:35.659201 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:35.662429 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:35.663147 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:35.663155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:35.663162 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:35.663331 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:35.663720 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:35.663726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:35.663729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:35.664261 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:35.664588 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:35.665038 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 03:03:35.665483 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":298,"threads_started":1}
I20260620 03:03:35.667239 17426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:35.751479 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 17303
2026-06-20T03:03:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64550 ms)
[----------] 30 tests from SecurityITest (295352 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:03:35 dist-test-slave-f1j0 krb5kdc[17488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:03:35 dist-test-slave-f1j0 krb5kdc[17488](info): set up 2 sockets
Jun 20 03:03:35 dist-test-slave-f1j0 krb5kdc[17488](info): commencing operation
krb5kdc: starting...
W20260620 03:03:36.861660 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.013s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:03:37 dist-test-slave-f1j0 krb5kdc[17488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:03:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:03:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:03:37.111387 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46143
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:03:37.443827 17501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:37.444134 17501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:37.444303 17501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:37.451665 17501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:03:37.451790 17501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:37.451875 17501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:37.451962 17501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:03:37.452045 17501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:03:37.462004 17501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:37.463361 17501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:37.464753 17501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:37.476662 17507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:37.476755 17509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:37.476662 17506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:37.477550 17501 server_base.cc:1034] running on GCE node
I20260620 03:03:37.478365 17501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:37.479820 17501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:37.481060 17501 hybrid_clock.cc:648] HybridClock initialized: now 1781924617481011 us; error 30 us; skew 500 ppm
I20260620 03:03:37.481402 17501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:03:37 dist-test-slave-f1j0 krb5kdc[17488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:03:37.484638 17501 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:03:37.486174 17501 webserver.cc:460] Webserver started at http://127.0.97.62:44413/ using document root <none> and password file <none>
I20260620 03:03:37.486843 17501 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:37.486943 17501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:37.487211 17501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:37.490372 17501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "70622be061a54c709b802e53425d5af9"
format_stamp: "Formatted at 2026-06-20 03:03:37 on dist-test-slave-f1j0"
server_key: "6eb2bab3d5c9ab63cf0299148a1e635b"
server_key_iv: "af8a88b3adb3bd6f071a4cbd7ef9b60c"
server_key_version: "encryptionkey@0"
I20260620 03:03:37.491168 17501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "70622be061a54c709b802e53425d5af9"
format_stamp: "Formatted at 2026-06-20 03:03:37 on dist-test-slave-f1j0"
server_key: "6eb2bab3d5c9ab63cf0299148a1e635b"
server_key_iv: "af8a88b3adb3bd6f071a4cbd7ef9b60c"
server_key_version: "encryptionkey@0"
I20260620 03:03:37.496771 17501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 03:03:37.500635 17516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:37.502233 17501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:03:37.502444 17501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "70622be061a54c709b802e53425d5af9"
format_stamp: "Formatted at 2026-06-20 03:03:37 on dist-test-slave-f1j0"
server_key: "6eb2bab3d5c9ab63cf0299148a1e635b"
server_key_iv: "af8a88b3adb3bd6f071a4cbd7ef9b60c"
server_key_version: "encryptionkey@0"
I20260620 03:03:37.502676 17501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:37.527537 17501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:37.532233 17501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:37.532558 17501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:37.550937 17501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46143
I20260620 03:03:37.550966 17567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46143 every 8 connection(s)
I20260620 03:03:37.552366 17501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:03:37.556293 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 17501
I20260620 03:03:37.556380 17568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:37.556519 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:03:37.557356 388 external_mini_cluster.cc:1428] Setting key 44989099ffe38149e528b33ea0344971
I20260620 03:03:37.571939 17568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9: Bootstrap starting.
Jun 20 03:03:37 dist-test-slave-f1j0 krb5kdc[17488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:03:37.576004 17568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:37.577333 17568 log.cc:824] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:37.580528 17568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9: No bootstrap required, opened a new log
I20260620 03:03:37.582808 17571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:37.559521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.0.97.62:46143)
0620 03:03:37.560347 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:37.560359 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:37.560400 (+ 41us) server_negotiation.cc:407] Connection header received
0620 03:03:37.562711 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:37.562734 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:37.563142 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:37.563482 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:37.565555 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:37.567069 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:37.569312 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:37.570102 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:37.570237 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:37.575286 (+ 5049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:37.575294 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:37.575309 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:37.575345 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:37.578659 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:37.579378 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:37.579391 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:37.579398 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:37.579566 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:37.579963 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:37.579970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:37.579973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:37.580484 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:37.580800 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:37.581313 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 03:03:37.581813 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":612,"thread_start_us":406,"threads_started":1}
I20260620 03:03:37.589659 17568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70622be061a54c709b802e53425d5af9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46143 } }
I20260620 03:03:37.590135 17568 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:37.590240 17568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70622be061a54c709b802e53425d5af9, State: Initialized, Role: FOLLOWER
I20260620 03:03:37.590834 17568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70622be061a54c709b802e53425d5af9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46143 } }
I20260620 03:03:37.591073 17568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:03:37.591212 17568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:03:37.591379 17568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:37.593787 17568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70622be061a54c709b802e53425d5af9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46143 } }
I20260620 03:03:37.594278 17568 leader_election.cc:302] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70622be061a54c709b802e53425d5af9; no voters:
I20260620 03:03:37.594821 17568 leader_election.cc:288] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:03:37.594995 17573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:03:37.595803 17573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9 [term 1 LEADER]: Becoming Leader. State: Replica: 70622be061a54c709b802e53425d5af9, State: Running, Role: LEADER
I20260620 03:03:37.596437 17573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70622be061a54c709b802e53425d5af9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46143 } }
I20260620 03:03:37.597312 17568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:03:37.599714 17574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70622be061a54c709b802e53425d5af9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70622be061a54c709b802e53425d5af9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46143 } } }
I20260620 03:03:37.600224 17574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9 [sys.catalog]: This master's current role is: LEADER
I20260620 03:03:37.600096 17575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70622be061a54c709b802e53425d5af9. Latest consensus state: current_term: 1 leader_uuid: "70622be061a54c709b802e53425d5af9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70622be061a54c709b802e53425d5af9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46143 } } }
I20260620 03:03:37.600418 17575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9 [sys.catalog]: This master's current role is: LEADER
I20260620 03:03:37.602208 17580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:03:37.607427 17580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:03:37.617069 17580 catalog_manager.cc:1347] Generated new cluster ID: 454809b7ec8a4aef974baaab48a7d658
I20260620 03:03:37.617199 17580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:03:37.639950 17580 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:03:37.642100 17580 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:03:37.653393 17580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9: Generated new TSK 0
I20260620 03:03:37.654599 17580 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:03:37.723735 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46143
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:03:38.060411 17594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:38.060745 17594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:38.060918 17594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:38.067961 17594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:38.068120 17594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:38.068307 17594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:03:38.078058 17594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:38.079586 17594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:38.080881 17594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:38.095340 17599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:38.095363 17600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:38.095285 17602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:38.096040 17594 server_base.cc:1034] running on GCE node
I20260620 03:03:38.096688 17594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:38.097909 17594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:38.099116 17594 hybrid_clock.cc:648] HybridClock initialized: now 1781924618099058 us; error 48 us; skew 500 ppm
I20260620 03:03:38.099440 17594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:03:38 dist-test-slave-f1j0 krb5kdc[17488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:03:38.102954 17594 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:03:38.104476 17594 webserver.cc:460] Webserver started at http://127.0.97.1:43955/ using document root <none> and password file <none>
I20260620 03:03:38.105139 17594 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:38.105235 17594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:38.105489 17594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:38.108639 17594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "4bdacd334d7749169627b1032e8a33dd"
format_stamp: "Formatted at 2026-06-20 03:03:38 on dist-test-slave-f1j0"
server_key: "de6f4f54f465426ac05473fc8402826e"
server_key_iv: "bb8ac1345c2d4f11e7dcc39e50b55958"
server_key_version: "encryptionkey@0"
I20260620 03:03:38.109385 17594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4bdacd334d7749169627b1032e8a33dd"
format_stamp: "Formatted at 2026-06-20 03:03:38 on dist-test-slave-f1j0"
server_key: "de6f4f54f465426ac05473fc8402826e"
server_key_iv: "bb8ac1345c2d4f11e7dcc39e50b55958"
server_key_version: "encryptionkey@0"
I20260620 03:03:38.115497 17594 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260620 03:03:38.120229 17609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:38.121678 17594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:03:38.121819 17594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "4bdacd334d7749169627b1032e8a33dd"
format_stamp: "Formatted at 2026-06-20 03:03:38 on dist-test-slave-f1j0"
server_key: "de6f4f54f465426ac05473fc8402826e"
server_key_iv: "bb8ac1345c2d4f11e7dcc39e50b55958"
server_key_version: "encryptionkey@0"
I20260620 03:03:38.121987 17594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:38.145838 17594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:38.149940 17594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:38.150195 17594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:38.151160 17594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:38.152490 17594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:38.152560 17594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:38.152643 17594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:38.152688 17594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:38.186496 17594 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38255
I20260620 03:03:38.186597 17721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38255 every 8 connection(s)
I20260620 03:03:38.188099 17594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:03:38.191804 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 17594
I20260620 03:03:38.192063 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:03:38.192937 388 external_mini_cluster.cc:1428] Setting key f445657ede4f6840ea7e59d6ae28a844
Jun 20 03:03:38 dist-test-slave-f1j0 krb5kdc[17488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:03:38.209821 17724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:38.190942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52755 (local address 127.0.97.62:46143)
0620 03:03:38.191467 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:38.191476 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:38.192522 (+ 1046us) server_negotiation.cc:407] Connection header received
0620 03:03:38.194915 (+ 2393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:38.194920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:38.195013 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:38.195127 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:38.196596 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:38.197473 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:38.199202 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:38.199611 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:38.199756 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:38.203590 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:38.203596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:38.203599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:38.203632 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:38.206412 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:38.207386 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:38.207389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:38.207391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:38.207580 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:38.208043 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:38.208049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:38.208052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:38.208306 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:38.208483 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:38.209170 (+ 687us) server_negotiation.cc:299] Negotiation successful
0620 03:03:38.209413 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":226,"threads_started":1}
I20260620 03:03:38.210781 17725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:38.191486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46143 (local address 127.0.97.1:52755)
0620 03:03:38.192340 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:38.192399 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:38.194620 (+ 2221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:38.195362 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:38.195370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:38.195868 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:03:38.196426 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:38.196443 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:38.197683 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:38.197688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:38.199052 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:38.199064 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:38.199841 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:38.199849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:38.200100 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:38.200784 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:03:38.200806 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:03:38.203313 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:03:38.206575 (+ 3262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:03:38.206587 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:03:38.206607 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:03:38.207284 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:03:38.207708 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:03:38.207713 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:03:38.207717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:03:38.207912 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:03:38.208436 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:03:38.208443 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:03:38.208798 (+ 355us) client_negotiation.cc:769] Sending connection context
0620 03:03:38.209070 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 03:03:38.209639 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":309,"threads_started":1}
I20260620 03:03:38.212487 17722 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46143
I20260620 03:03:38.212817 17722 heartbeater.cc:464] Registering TS with master...
I20260620 03:03:38.213529 17722 heartbeater.cc:511] Master 127.0.97.62:46143 requested a full tablet report, sending...
I20260620 03:03:38.215775 17533 ts_manager.cc:194] Registered new tserver with Master: 4bdacd334d7749169627b1032e8a33dd (127.0.97.1:38255)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 03:03:38.217283 17533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:03:38.247152 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46143
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:03:38.573071 17728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:38.573343 17728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:38.573436 17728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:38.580915 17728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:38.581027 17728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:38.581135 17728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:03:38.591025 17728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:38.592250 17728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:38.593509 17728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:38.606715 17734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:38.606822 17733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:38.606673 17736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:38.607640 17728 server_base.cc:1034] running on GCE node
I20260620 03:03:38.608294 17728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:38.609301 17728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:38.610550 17728 hybrid_clock.cc:648] HybridClock initialized: now 1781924618610495 us; error 66 us; skew 500 ppm
I20260620 03:03:38.610869 17728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:03:38 dist-test-slave-f1j0 krb5kdc[17488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:03:38.613929 17728 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:03:38.615272 17728 webserver.cc:460] Webserver started at http://127.0.97.2:40153/ using document root <none> and password file <none>
I20260620 03:03:38.615958 17728 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:38.616101 17728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:38.616453 17728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:38.619469 17728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "6c81c403b41e4b4b86531987a46980d9"
format_stamp: "Formatted at 2026-06-20 03:03:38 on dist-test-slave-f1j0"
server_key: "ce28976781d35ee8bebae90d9579e94a"
server_key_iv: "900390754910b806198b1c59e9370598"
server_key_version: "encryptionkey@0"
I20260620 03:03:38.620242 17728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6c81c403b41e4b4b86531987a46980d9"
format_stamp: "Formatted at 2026-06-20 03:03:38 on dist-test-slave-f1j0"
server_key: "ce28976781d35ee8bebae90d9579e94a"
server_key_iv: "900390754910b806198b1c59e9370598"
server_key_version: "encryptionkey@0"
I20260620 03:03:38.625239 17728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 03:03:38.628916 17743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:38.630630 17728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 03:03:38.630808 17728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "6c81c403b41e4b4b86531987a46980d9"
format_stamp: "Formatted at 2026-06-20 03:03:38 on dist-test-slave-f1j0"
server_key: "ce28976781d35ee8bebae90d9579e94a"
server_key_iv: "900390754910b806198b1c59e9370598"
server_key_version: "encryptionkey@0"
I20260620 03:03:38.631029 17728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:38.645161 17728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:38.649149 17728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:38.649492 17728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:38.650488 17728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:38.651765 17728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:38.651837 17728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:38.651917 17728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:38.651965 17728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:38.685694 17728 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34107
I20260620 03:03:38.685813 17855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34107 every 8 connection(s)
I20260620 03:03:38.687060 17728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:03:38.690287 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 17728
I20260620 03:03:38.690485 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:03:38.691114 388 external_mini_cluster.cc:1428] Setting key e402bd4dabf974c29490c327bf53c360
Jun 20 03:03:38 dist-test-slave-f1j0 krb5kdc[17488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:03:38.707994 17724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:38.689729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58901 (local address 127.0.97.62:46143)
0620 03:03:38.689979 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:38.689983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:38.691470 (+ 1487us) server_negotiation.cc:407] Connection header received
0620 03:03:38.693551 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:38.693557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:38.693667 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:38.693786 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:38.695262 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:38.695970 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:38.697674 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:38.698109 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:38.698242 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:38.702105 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:38.702111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:38.702114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:38.702145 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:38.704840 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:38.705627 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:38.705632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:38.705634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:38.705750 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:38.706229 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:38.706240 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:38.706243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:38.706515 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:38.706670 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:38.707398 (+ 728us) server_negotiation.cc:299] Negotiation successful
0620 03:03:38.707637 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 03:03:38.708966 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:38.690385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46143 (local address 127.0.97.2:58901)
0620 03:03:38.691277 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:38.691331 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:38.693287 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:38.694033 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:38.694042 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:38.694536 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:03:38.695095 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:38.695113 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:38.696122 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:38.696127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:38.697502 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:38.697533 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:38.698345 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:38.698352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:38.698682 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:38.699406 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:03:38.699430 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:03:38.701874 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:03:38.704984 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:03:38.705003 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:03:38.705021 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:03:38.705470 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:03:38.705860 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:03:38.705866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:03:38.705870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:03:38.706107 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:03:38.706641 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:03:38.706648 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:03:38.707021 (+ 373us) client_negotiation.cc:769] Sending connection context
0620 03:03:38.707303 (+ 282us) client_negotiation.cc:240] Negotiation successful
0620 03:03:38.707837 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":313,"threads_started":1}
I20260620 03:03:38.710471 17856 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46143
I20260620 03:03:38.710844 17856 heartbeater.cc:464] Registering TS with master...
I20260620 03:03:38.711778 17856 heartbeater.cc:511] Master 127.0.97.62:46143 requested a full tablet report, sending...
I20260620 03:03:38.713541 17533 ts_manager.cc:194] Registered new tserver with Master: 6c81c403b41e4b4b86531987a46980d9 (127.0.97.2:34107)
I20260620 03:03:38.714406 17533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58901
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:03:38.744994 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46143
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:03:39.072551 17861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:39.072834 17861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:39.072929 17861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:39.080276 17861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:39.080380 17861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:39.080499 17861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:03:39.090337 17861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:39.091662 17861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:39.092980 17861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:39.106769 17867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:39.106840 17866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:39.106738 17869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:39.107610 17861 server_base.cc:1034] running on GCE node
I20260620 03:03:39.108249 17861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:39.109190 17861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:39.110416 17861 hybrid_clock.cc:648] HybridClock initialized: now 1781924619110390 us; error 87 us; skew 500 ppm
I20260620 03:03:39.110728 17861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:03:39 dist-test-slave-f1j0 krb5kdc[17488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:03:39.114096 17861 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:03:39.115536 17861 webserver.cc:460] Webserver started at http://127.0.97.3:39877/ using document root <none> and password file <none>
I20260620 03:03:39.116237 17861 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:39.116379 17861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:39.116714 17861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:39.119591 17861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "6f09d32d75aa4ad89a05432caeadf42e"
format_stamp: "Formatted at 2026-06-20 03:03:39 on dist-test-slave-f1j0"
server_key: "6867a301f3e36ea966c380fd4d02628c"
server_key_iv: "47c0910dd764c83485afd25095de0901"
server_key_version: "encryptionkey@0"
I20260620 03:03:39.120342 17861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6f09d32d75aa4ad89a05432caeadf42e"
format_stamp: "Formatted at 2026-06-20 03:03:39 on dist-test-slave-f1j0"
server_key: "6867a301f3e36ea966c380fd4d02628c"
server_key_iv: "47c0910dd764c83485afd25095de0901"
server_key_version: "encryptionkey@0"
I20260620 03:03:39.125358 17861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 03:03:39.128940 17876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:39.130723 17861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:03:39.130927 17861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "6f09d32d75aa4ad89a05432caeadf42e"
format_stamp: "Formatted at 2026-06-20 03:03:39 on dist-test-slave-f1j0"
server_key: "6867a301f3e36ea966c380fd4d02628c"
server_key_iv: "47c0910dd764c83485afd25095de0901"
server_key_version: "encryptionkey@0"
I20260620 03:03:39.131210 17861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:39.155594 17861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:39.160020 17861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:39.160267 17861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:39.161187 17861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:39.162510 17861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:39.162582 17861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:39.162664 17861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:39.162708 17861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:39.196177 17861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39983
I20260620 03:03:39.196233 17988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39983 every 8 connection(s)
I20260620 03:03:39.197422 17861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:03:39.198441 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 17861
I20260620 03:03:39.198666 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:03:39.199155 388 external_mini_cluster.cc:1428] Setting key 424d892bd9c944834ce9aad7672848a6
Jun 20 03:03:39 dist-test-slave-f1j0 krb5kdc[17488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:03:39.217711 17724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:39.200308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60817 (local address 127.0.97.62:46143)
0620 03:03:39.200509 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:39.200515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:39.201870 (+ 1355us) server_negotiation.cc:407] Connection header received
0620 03:03:39.203847 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:39.203852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:39.203916 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:39.203993 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:39.205448 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.206380 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.207956 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.208333 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.208421 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:39.211943 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:39.211947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:39.211950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:39.211982 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:39.215069 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:39.215699 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:39.215702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:39.215704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:39.215823 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:39.216191 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:39.216194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:39.216196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:39.216465 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:39.216573 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:39.217219 (+ 646us) server_negotiation.cc:299] Negotiation successful
0620 03:03:39.217387 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:03:39.218583 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:39.200821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46143 (local address 127.0.97.3:60817)
0620 03:03:39.201710 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:39.201755 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:39.203591 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:39.204243 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:39.204251 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:39.204741 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:03:39.205300 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:39.205316 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.206520 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.206524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:39.207824 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:39.207836 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.208552 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.208564 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:39.208765 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:39.209410 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:03:39.209432 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:03:39.211775 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:03:39.215184 (+ 3409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:03:39.215190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:03:39.215201 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:03:39.215601 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:03:39.215912 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:03:39.215915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:03:39.215917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:03:39.216101 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:03:39.216571 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:03:39.216576 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:03:39.216855 (+ 279us) client_negotiation.cc:769] Sending connection context
0620 03:03:39.217086 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 03:03:39.217443 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":336,"threads_started":1}
I20260620 03:03:39.219856 17989 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46143
I20260620 03:03:39.220156 17989 heartbeater.cc:464] Registering TS with master...
I20260620 03:03:39.220554 17722 heartbeater.cc:503] Master 127.0.97.62:46143 was elected leader, sending a full tablet report...
I20260620 03:03:39.220937 17989 heartbeater.cc:511] Master 127.0.97.62:46143 requested a full tablet report, sending...
I20260620 03:03:39.222366 17533 ts_manager.cc:194] Registered new tserver with Master: 6f09d32d75aa4ad89a05432caeadf42e (127.0.97.3:39983)
I20260620 03:03:39.223109 17533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60817
I20260620 03:03:39.226979 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:03:39.243412 17724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:39.231417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.0.97.62:46143)
0620 03:03:39.231636 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:39.231643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:39.232033 (+ 390us) server_negotiation.cc:407] Connection header received
0620 03:03:39.232159 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:39.232163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:39.232220 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:39.232300 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:39.233560 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.234298 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.235592 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.236071 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.236167 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:39.238602 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:39.238607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:39.238610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:39.238639 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:39.241231 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:39.241873 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:39.241876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:39.241879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:39.241988 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:39.242276 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:39.242279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:39.242281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:39.242581 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:39.242770 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:39.243032 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 03:03:39.243198 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:03:39.247411 17533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:03:39.254315 17533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:03:39.291855 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:39.266165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34107 (local address 127.0.0.1:46104)
0620 03:03:39.278121 (+ 11956us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:39.278188 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:39.278420 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:39.284291 (+ 5871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:39.284300 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:39.284331 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:39.284698 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:39.284713 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.286762 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.286768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:39.289129 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:39.289143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.291081 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.291087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:39.291236 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:39.291264 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 03:03:39.291339 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 03:03:39.291453 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11570,"mutex_wait_us":66,"thread_start_us":15201,"threads_started":1}
I20260620 03:03:39.291887 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:39.279667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39983 (local address 127.0.0.1:51902)
0620 03:03:39.283275 (+ 3608us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:39.283296 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:39.283421 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:39.284826 (+ 1405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:39.284830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:39.284860 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:39.285249 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:39.285259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.286915 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.286920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:39.288982 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:39.288994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.290754 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.290761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:39.290971 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:39.291041 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 03:03:39.291160 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 03:03:39.291355 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3463,"mutex_wait_us":28,"thread_start_us":3322,"threads_started":1}
I20260620 03:03:39.292454 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:39.280017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51902 (local address 127.0.97.3:39983)
0620 03:03:39.284212 (+ 4195us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:39.284223 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:39.284262 (+ 39us) server_negotiation.cc:407] Connection header received
0620 03:03:39.284341 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:39.284346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:39.284528 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:39.284736 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:39.285484 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.286763 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.289202 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.290511 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.291335 (+ 824us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:39.291577 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:39.291655 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 03:03:39.291817 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4016,"thread_start_us":3855,"threads_started":1}
I20260620 03:03:39.292680 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:39.266279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46104 (local address 127.0.97.2:34107)
0620 03:03:39.283551 (+ 17272us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:39.283561 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:39.283589 (+ 28us) server_negotiation.cc:407] Connection header received
0620 03:03:39.283683 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:39.283694 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:39.284022 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:39.284919 (+ 897us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:39.285138 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.286607 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.289312 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.290837 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.291004 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:39.291217 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:39.292159 (+ 942us) server_negotiation.cc:299] Negotiation successful
0620 03:03:39.292347 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17091,"thread_start_us":17226,"threads_started":1}
I20260620 03:03:39.292892 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:39.266264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38255 (local address 127.0.0.1:55638)
0620 03:03:39.281092 (+ 14828us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:39.281117 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:39.281263 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:39.285369 (+ 4106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:39.285377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:39.285406 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:39.285803 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:39.285815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.287716 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.287722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:39.289971 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:39.289985 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.292089 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.292095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:39.292272 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:39.292306 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 03:03:39.292381 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 03:03:39.292519 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14665,"mutex_wait_us":1,"thread_start_us":12396,"threads_started":1}
I20260620 03:03:39.295703 17924 tablet_service.cc:1467] Processing CreateTablet for tablet 2bec627c08e949e390cc11bcc790b085 (DEFAULT_TABLE table=test-table [id=65aa7472af4040ca922018d88b26d23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:03:39.295707 17791 tablet_service.cc:1467] Processing CreateTablet for tablet 2bec627c08e949e390cc11bcc790b085 (DEFAULT_TABLE table=test-table [id=65aa7472af4040ca922018d88b26d23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:03:39.296643 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:39.267109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55638 (local address 127.0.97.1:38255)
0620 03:03:39.284570 (+ 17461us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:39.284581 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:39.284610 (+ 29us) server_negotiation.cc:407] Connection header received
0620 03:03:39.284919 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:39.284926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:39.285162 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:39.285358 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:39.285975 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.287557 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.290161 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.291594 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.291712 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:39.291916 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:39.293216 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0620 03:03:39.293378 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17279,"thread_start_us":17844,"threads_started":1}
I20260620 03:03:39.297124 17924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bec627c08e949e390cc11bcc790b085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:39.297134 17791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bec627c08e949e390cc11bcc790b085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:39.300336 17657 tablet_service.cc:1467] Processing CreateTablet for tablet 2bec627c08e949e390cc11bcc790b085 (DEFAULT_TABLE table=test-table [id=65aa7472af4040ca922018d88b26d23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:03:39.301689 17657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bec627c08e949e390cc11bcc790b085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:39.310318 18007 tablet_bootstrap.cc:492] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e: Bootstrap starting.
I20260620 03:03:39.313874 18007 tablet_bootstrap.cc:654] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:39.315030 18007 log.cc:824] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:39.315979 18008 tablet_bootstrap.cc:492] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9: Bootstrap starting.
I20260620 03:03:39.317368 18010 tablet_bootstrap.cc:492] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd: Bootstrap starting.
I20260620 03:03:39.320132 18008 tablet_bootstrap.cc:654] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:39.320437 18010 tablet_bootstrap.cc:654] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:39.321137 18007 tablet_bootstrap.cc:492] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e: No bootstrap required, opened a new log
I20260620 03:03:39.321431 18010 log.cc:824] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:39.321462 18007 ts_tablet_manager.cc:1397] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260620 03:03:39.321496 18008 log.cc:824] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:39.324244 18008 tablet_bootstrap.cc:492] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9: No bootstrap required, opened a new log
I20260620 03:03:39.324517 18008 ts_tablet_manager.cc:1397] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 03:03:39.324617 18010 tablet_bootstrap.cc:492] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd: No bootstrap required, opened a new log
I20260620 03:03:39.324890 18010 ts_tablet_manager.cc:1397] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 03:03:39.332329 18008 raft_consensus.cc:348] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c81c403b41e4b4b86531987a46980d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34107 } } peers { permanent_uuid: "4bdacd334d7749169627b1032e8a33dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } } peers { permanent_uuid: "6f09d32d75aa4ad89a05432caeadf42e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39983 } }
I20260620 03:03:39.332712 18008 raft_consensus.cc:374] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:39.332809 18008 raft_consensus.cc:729] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c81c403b41e4b4b86531987a46980d9, State: Initialized, Role: FOLLOWER
I20260620 03:03:39.333434 18008 consensus_queue.cc:260] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c81c403b41e4b4b86531987a46980d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34107 } } peers { permanent_uuid: "4bdacd334d7749169627b1032e8a33dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } } peers { permanent_uuid: "6f09d32d75aa4ad89a05432caeadf42e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39983 } }
I20260620 03:03:39.333451 18007 raft_consensus.cc:348] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c81c403b41e4b4b86531987a46980d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34107 } } peers { permanent_uuid: "4bdacd334d7749169627b1032e8a33dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } } peers { permanent_uuid: "6f09d32d75aa4ad89a05432caeadf42e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39983 } }
I20260620 03:03:39.333951 18007 raft_consensus.cc:374] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:39.334056 18007 raft_consensus.cc:729] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f09d32d75aa4ad89a05432caeadf42e, State: Initialized, Role: FOLLOWER
I20260620 03:03:39.334688 18007 consensus_queue.cc:260] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c81c403b41e4b4b86531987a46980d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34107 } } peers { permanent_uuid: "4bdacd334d7749169627b1032e8a33dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } } peers { permanent_uuid: "6f09d32d75aa4ad89a05432caeadf42e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39983 } }
I20260620 03:03:39.336140 17989 heartbeater.cc:503] Master 127.0.97.62:46143 was elected leader, sending a full tablet report...
I20260620 03:03:39.336745 18007 ts_tablet_manager.cc:1428] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e: Time spent starting tablet: real 0.015s user 0.014s sys 0.003s
I20260620 03:03:39.338220 17856 heartbeater.cc:503] Master 127.0.97.62:46143 was elected leader, sending a full tablet report...
I20260620 03:03:39.338728 18008 ts_tablet_manager.cc:1428] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260620 03:03:39.337559 18010 raft_consensus.cc:348] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c81c403b41e4b4b86531987a46980d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34107 } } peers { permanent_uuid: "4bdacd334d7749169627b1032e8a33dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } } peers { permanent_uuid: "6f09d32d75aa4ad89a05432caeadf42e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39983 } }
I20260620 03:03:39.339850 18010 raft_consensus.cc:374] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:39.339974 18010 raft_consensus.cc:729] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bdacd334d7749169627b1032e8a33dd, State: Initialized, Role: FOLLOWER
I20260620 03:03:39.340610 18010 consensus_queue.cc:260] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c81c403b41e4b4b86531987a46980d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34107 } } peers { permanent_uuid: "4bdacd334d7749169627b1032e8a33dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } } peers { permanent_uuid: "6f09d32d75aa4ad89a05432caeadf42e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39983 } }
I20260620 03:03:39.343966 18010 ts_tablet_manager.cc:1428] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd: Time spent starting tablet: real 0.019s user 0.014s sys 0.000s
W20260620 03:03:39.439139 17857 tablet.cc:2369] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:03:39.440475 17723 tablet.cc:2369] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:03:39.449306 17990 tablet.cc:2369] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:03:39.598817 18013 raft_consensus.cc:482] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:03:39.599184 18013 raft_consensus.cc:504] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c81c403b41e4b4b86531987a46980d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34107 } } peers { permanent_uuid: "4bdacd334d7749169627b1032e8a33dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } } peers { permanent_uuid: "6f09d32d75aa4ad89a05432caeadf42e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39983 } }
I20260620 03:03:39.600839 18013 leader_election.cc:288] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bdacd334d7749169627b1032e8a33dd (127.0.97.1:38255), 6f09d32d75aa4ad89a05432caeadf42e (127.0.97.3:39983)
I20260620 03:03:39.608498 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:39.602452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50397 (local address 127.0.97.3:39983)
0620 03:03:39.602676 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:39.602683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:39.602875 (+ 192us) server_negotiation.cc:407] Connection header received
0620 03:03:39.603004 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:39.603012 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:39.603118 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:39.603212 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:39.603880 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.604638 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.606627 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.607517 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.607666 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:39.607780 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:39.608093 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 03:03:39.608219 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 03:03:39.608506 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:39.601355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33415 (local address 127.0.97.1:38255)
0620 03:03:39.601558 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:39.601562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:39.602471 (+ 909us) server_negotiation.cc:407] Connection header received
0620 03:03:39.602566 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:39.602571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:39.602649 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:39.602756 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:39.603568 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.604538 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.606678 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.607536 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.607677 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:39.607787 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:39.608137 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 03:03:39.608255 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:03:39.608656 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:39.602174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39983 (local address 127.0.97.2:50397)
0620 03:03:39.602773 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:39.602787 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:39.602883 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:39.603234 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:39.603237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:39.603255 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:39.603658 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:39.603673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.604778 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.604785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:39.606473 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:39.606488 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.607744 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.607751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:39.607932 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:39.607957 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 03:03:39.608039 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 03:03:39.608188 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":17,"thread_start_us":319,"threads_started":1}
I20260620 03:03:39.608909 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:39.601288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38255 (local address 127.0.97.2:33415)
0620 03:03:39.602026 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:39.602048 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:39.602201 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:39.602939 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:39.602947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:39.602984 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:39.603383 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:39.603394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.604671 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.604679 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:39.606496 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:39.606508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.607780 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.607787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:39.607946 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:39.607976 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 03:03:39.608086 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 03:03:39.608215 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":96,"thread_start_us":419,"threads_started":1}
I20260620 03:03:39.609791 17944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bec627c08e949e390cc11bcc790b085" candidate_uuid: "6c81c403b41e4b4b86531987a46980d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f09d32d75aa4ad89a05432caeadf42e" is_pre_election: true
I20260620 03:03:39.610023 17677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bec627c08e949e390cc11bcc790b085" candidate_uuid: "6c81c403b41e4b4b86531987a46980d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bdacd334d7749169627b1032e8a33dd" is_pre_election: true
I20260620 03:03:39.610409 17944 raft_consensus.cc:2454] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c81c403b41e4b4b86531987a46980d9 in term 0.
I20260620 03:03:39.610546 17677 raft_consensus.cc:2454] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c81c403b41e4b4b86531987a46980d9 in term 0.
I20260620 03:03:39.611090 17745 leader_election.cc:302] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c81c403b41e4b4b86531987a46980d9, 6f09d32d75aa4ad89a05432caeadf42e; no voters:
I20260620 03:03:39.611410 18013 raft_consensus.cc:2781] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:03:39.611536 18013 raft_consensus.cc:482] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:03:39.611678 18013 raft_consensus.cc:3037] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:39.613905 18013 raft_consensus.cc:504] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c81c403b41e4b4b86531987a46980d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34107 } } peers { permanent_uuid: "4bdacd334d7749169627b1032e8a33dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } } peers { permanent_uuid: "6f09d32d75aa4ad89a05432caeadf42e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39983 } }
I20260620 03:03:39.614477 18013 leader_election.cc:288] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4bdacd334d7749169627b1032e8a33dd (127.0.97.1:38255), 6f09d32d75aa4ad89a05432caeadf42e (127.0.97.3:39983)
I20260620 03:03:39.614972 17677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bec627c08e949e390cc11bcc790b085" candidate_uuid: "6c81c403b41e4b4b86531987a46980d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bdacd334d7749169627b1032e8a33dd"
I20260620 03:03:39.615044 17944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bec627c08e949e390cc11bcc790b085" candidate_uuid: "6c81c403b41e4b4b86531987a46980d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f09d32d75aa4ad89a05432caeadf42e"
I20260620 03:03:39.615186 17677 raft_consensus.cc:3037] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:39.615224 17944 raft_consensus.cc:3037] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:39.618436 17944 raft_consensus.cc:2454] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c81c403b41e4b4b86531987a46980d9 in term 1.
I20260620 03:03:39.618436 17677 raft_consensus.cc:2454] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c81c403b41e4b4b86531987a46980d9 in term 1.
I20260620 03:03:39.618853 17745 leader_election.cc:302] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bdacd334d7749169627b1032e8a33dd, 6c81c403b41e4b4b86531987a46980d9; no voters:
I20260620 03:03:39.619163 18013 raft_consensus.cc:2781] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:03:39.619632 18013 raft_consensus.cc:686] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [term 1 LEADER]: Becoming Leader. State: Replica: 6c81c403b41e4b4b86531987a46980d9, State: Running, Role: LEADER
I20260620 03:03:39.620098 18013 consensus_queue.cc:237] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c81c403b41e4b4b86531987a46980d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34107 } } peers { permanent_uuid: "4bdacd334d7749169627b1032e8a33dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } } peers { permanent_uuid: "6f09d32d75aa4ad89a05432caeadf42e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39983 } }
I20260620 03:03:39.625123 17533 catalog_manager.cc:5495] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c81c403b41e4b4b86531987a46980d9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "6c81c403b41e4b4b86531987a46980d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c81c403b41e4b4b86531987a46980d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bdacd334d7749169627b1032e8a33dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f09d32d75aa4ad89a05432caeadf42e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39983 } health_report { overall_health: UNKNOWN } } }
I20260620 03:03:39.642625 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:39.637296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.0.97.2:34107)
0620 03:03:39.637502 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:39.637520 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:39.637540 (+ 20us) server_negotiation.cc:407] Connection header received
0620 03:03:39.637798 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:39.637802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:39.637877 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:39.638020 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:03:39.638396 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.639095 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.640491 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.640888 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.641019 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:39.641505 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:03:39.642053 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:03:39.642211 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:39.642265 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 03:03:39.642341 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 03:03:39.650683 17944 raft_consensus.cc:1261] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e [term 1 FOLLOWER]: Refusing update from remote peer 6c81c403b41e4b4b86531987a46980d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:03:39.650859 17677 raft_consensus.cc:1261] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd [term 1 FOLLOWER]: Refusing update from remote peer 6c81c403b41e4b4b86531987a46980d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:03:39.651678 18018 consensus_queue.cc:1035] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f09d32d75aa4ad89a05432caeadf42e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:03:39.652042 18013 consensus_queue.cc:1035] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bdacd334d7749169627b1032e8a33dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:03:39.667946 18022 mvcc.cc:205] Tried to move back new op lower bound from 7298763242079141888 to 7298763241967562752. Current Snapshot: MvccSnapshot[applied={T|T < 7298763242079141888}]
I20260620 03:03:39.678009 17533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:03:39.678289 17533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:03:39.682051 17533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70622be061a54c709b802e53425d5af9: Sending DeleteTablet for 3 replicas of tablet 2bec627c08e949e390cc11bcc790b085
I20260620 03:03:39.683266 17791 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bec627c08e949e390cc11bcc790b085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:03:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46104
I20260620 03:03:39.683326 17924 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bec627c08e949e390cc11bcc790b085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:03:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51902
I20260620 03:03:39.683518 17657 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bec627c08e949e390cc11bcc790b085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:03:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55638
I20260620 03:03:39.686023 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 17594
I20260620 03:03:39.694525 18032 tablet_replica.cc:330] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd: stopping tablet replica
I20260620 03:03:39.694993 18032 raft_consensus.cc:2229] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:03:39.695523 18032 raft_consensus.cc:2258] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:03:39.696508 18030 tablet_replica.cc:330] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e: stopping tablet replica
I20260620 03:03:39.696910 18030 raft_consensus.cc:2229] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:03:39.697391 18030 raft_consensus.cc:2258] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:03:39.699167 18032 ts_tablet_manager.cc:1905] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:03:39.703855 18030 ts_tablet_manager.cc:1905] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:03:39.706296 18031 tablet_replica.cc:330] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9: stopping tablet replica
I20260620 03:03:39.706710 18031 raft_consensus.cc:2229] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:03:39.707326 18031 raft_consensus.cc:2258] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 03:03:39 dist-test-slave-f1j0 krb5kdc[17488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:03:39.713737 18032 ts_tablet_manager.cc:1918] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:03:39.714040 18032 log.cc:1196] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/2bec627c08e949e390cc11bcc790b085
I20260620 03:03:39.714452 18030 ts_tablet_manager.cc:1918] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:03:39.714635 18030 log.cc:1196] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/2bec627c08e949e390cc11bcc790b085
I20260620 03:03:39.715132 18030 ts_tablet_manager.cc:1939] T 2bec627c08e949e390cc11bcc790b085 P 6f09d32d75aa4ad89a05432caeadf42e: Deleting consensus metadata
I20260620 03:03:39.715296 18032 ts_tablet_manager.cc:1939] T 2bec627c08e949e390cc11bcc790b085 P 4bdacd334d7749169627b1032e8a33dd: Deleting consensus metadata
I20260620 03:03:39.717671 17518 catalog_manager.cc:4841] TS 4bdacd334d7749169627b1032e8a33dd (127.0.97.1:38255): tablet 2bec627c08e949e390cc11bcc790b085 (table test-table [id=65aa7472af4040ca922018d88b26d23a]) successfully deleted
I20260620 03:03:39.718293 17518 catalog_manager.cc:4841] TS 6f09d32d75aa4ad89a05432caeadf42e (127.0.97.3:39983): tablet 2bec627c08e949e390cc11bcc790b085 (table test-table [id=65aa7472af4040ca922018d88b26d23a]) successfully deleted
I20260620 03:03:39.718981 18031 ts_tablet_manager.cc:1905] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:03:39.721138 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:39.688001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55648 (local address 127.0.97.1:38255)
0620 03:03:39.688239 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:39.688248 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:39.701009 (+ 12761us) server_negotiation.cc:407] Connection header received
0620 03:03:39.701646 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:39.701656 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:39.701769 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:39.701863 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:39.704494 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.706048 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.707897 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.708307 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.708452 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:39.713842 (+ 5390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:39.713858 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:39.713879 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:39.713919 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:39.717932 (+ 4013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:39.718862 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:39.718871 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:39.718875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:39.719052 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:39.719525 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:39.719533 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:39.719538 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:39.719889 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:39.720201 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:39.720509 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 03:03:39.720742 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 03:03:39.721846 17717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:39.726545 18031 ts_tablet_manager.cc:1918] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:03:39.726728 18031 log.cc:1196] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/2bec627c08e949e390cc11bcc790b085
I20260620 03:03:39.727304 18031 ts_tablet_manager.cc:1939] T 2bec627c08e949e390cc11bcc790b085 P 6c81c403b41e4b4b86531987a46980d9: Deleting consensus metadata
I20260620 03:03:39.729128 17517 catalog_manager.cc:4841] TS 6c81c403b41e4b4b86531987a46980d9 (127.0.97.2:34107): tablet 2bec627c08e949e390cc11bcc790b085 (table test-table [id=65aa7472af4040ca922018d88b26d23a]) successfully deleted
I20260620 03:03:39.817036 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 17594
I20260620 03:03:39.837990 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 17728
Jun 20 03:03:39 dist-test-slave-f1j0 krb5kdc[17488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:03:39.854744 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:39.838808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46110 (local address 127.0.97.2:34107)
0620 03:03:39.838998 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:39.839004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:39.839024 (+ 20us) server_negotiation.cc:407] Connection header received
0620 03:03:39.839275 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:39.839282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:39.839384 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:39.839506 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:39.840691 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.841915 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.843136 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.843823 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.843955 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:39.847968 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:39.847986 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:39.848007 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:39.848059 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:39.851918 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:39.852615 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:39.852624 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:39.852628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:39.852789 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:39.853117 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:39.853122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:39.853126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:39.853623 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:39.853956 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:39.854096 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 03:03:39.854313 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 03:03:39.855433 17851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:39.968189 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 17728
I20260620 03:03:39.986711 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 17861
Jun 20 03:03:39 dist-test-slave-f1j0 krb5kdc[17488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:03:40.001561 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:39.987471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51904 (local address 127.0.97.3:39983)
0620 03:03:39.987649 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:39.987655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:39.987731 (+ 76us) server_negotiation.cc:407] Connection header received
0620 03:03:39.987968 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:39.987974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:39.988038 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:39.988144 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:39.989427 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.990222 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.991554 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:39.991943 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:39.992081 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:39.996386 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:39.996398 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:39.996415 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:39.996458 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:39.999247 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:39.999891 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:39.999897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:39.999900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:40.000038 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:40.000340 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:40.000343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:40.000345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:40.000688 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:40.000899 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:40.001116 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 03:03:40.001273 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 03:03:40.002336 17984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:40.093703 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 17861
I20260620 03:03:40.110564 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 17501
I20260620 03:03:40.111723 17563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:40.228231 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 17501
2026-06-20T03:03:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4479 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T03:03:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:03:40Z Disabled control of system clock
I20260620 03:03:40.304178 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39649
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 03:03:40.633446 18041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:40.633759 18041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:40.633842 18041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:40.641052 18041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:03:40.641180 18041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:40.641223 18041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:40.641274 18041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:03:40.641311 18041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:03:40.650712 18041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39649
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:40.652019 18041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:40.653797 18041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:40.665782 18049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:40.665855 18046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:40.665855 18047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:40.666498 18041 server_base.cc:1034] running on GCE node
I20260620 03:03:40.667307 18041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:40.668692 18041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:40.669965 18041 hybrid_clock.cc:648] HybridClock initialized: now 1781924620669914 us; error 62 us; skew 500 ppm
I20260620 03:03:40.670367 18041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:03:40.676599 18041 webserver.cc:460] Webserver started at http://127.0.97.62:45581/ using document root <none> and password file <none>
I20260620 03:03:40.677321 18041 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:40.677456 18041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:40.677838 18041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:40.680763 18041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "310bbad5f78b477fb5f7269797f74086"
format_stamp: "Formatted at 2026-06-20 03:03:40 on dist-test-slave-f1j0"
server_key: "d736a29447e8d987454335f45cb13dfe"
server_key_iv: "ab601ec79660d2a194519804cdb4d51d"
server_key_version: "encryptionkey@0"
I20260620 03:03:40.681561 18041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "310bbad5f78b477fb5f7269797f74086"
format_stamp: "Formatted at 2026-06-20 03:03:40 on dist-test-slave-f1j0"
server_key: "d736a29447e8d987454335f45cb13dfe"
server_key_iv: "ab601ec79660d2a194519804cdb4d51d"
server_key_version: "encryptionkey@0"
I20260620 03:03:40.686683 18041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 03:03:40.690274 18055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:40.691570 18041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:03:40.691751 18041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "310bbad5f78b477fb5f7269797f74086"
format_stamp: "Formatted at 2026-06-20 03:03:40 on dist-test-slave-f1j0"
server_key: "d736a29447e8d987454335f45cb13dfe"
server_key_iv: "ab601ec79660d2a194519804cdb4d51d"
server_key_version: "encryptionkey@0"
I20260620 03:03:40.691959 18041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:40.725432 18041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:40.726449 18041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:40.726758 18041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:40.745325 18041 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39649
I20260620 03:03:40.745358 18106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39649 every 8 connection(s)
I20260620 03:03:40.746712 18041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:03:40.750442 18107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:40.756935 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 18041
I20260620 03:03:40.757120 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:03:40.757721 388 external_mini_cluster.cc:1428] Setting key fd1c88be6dc2f3ad6f691fde769b17d4
I20260620 03:03:40.761752 18107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086: Bootstrap starting.
I20260620 03:03:40.764838 18107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:40.765928 18107 log.cc:824] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:40.769021 18107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086: No bootstrap required, opened a new log
I20260620 03:03:40.771821 18110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:40.759723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39604 (local address 127.0.97.62:39649)
0620 03:03:40.760672 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:40.760682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:40.760722 (+ 40us) server_negotiation.cc:407] Connection header received
0620 03:03:40.762484 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:40.762506 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:40.762945 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:40.763336 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:40.763884 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:40.765198 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:40.766815 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:40.767709 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:40.767900 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:40.768925 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:40.768935 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:40.768952 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:40.769754 (+ 802us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:40.769860 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:40.769893 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:40.770034 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:40.770240 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 03:03:40.770562 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":306,"threads_started":1}
I20260620 03:03:40.778146 18107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310bbad5f78b477fb5f7269797f74086" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39649 } }
I20260620 03:03:40.778462 18107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:40.778545 18107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310bbad5f78b477fb5f7269797f74086, State: Initialized, Role: FOLLOWER
I20260620 03:03:40.779062 18107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310bbad5f78b477fb5f7269797f74086" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39649 } }
I20260620 03:03:40.779263 18107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:03:40.779352 18107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:03:40.779469 18107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:40.781543 18107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310bbad5f78b477fb5f7269797f74086" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39649 } }
I20260620 03:03:40.782053 18107 leader_election.cc:302] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 310bbad5f78b477fb5f7269797f74086; no voters:
I20260620 03:03:40.782579 18107 leader_election.cc:288] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:03:40.782749 18112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:03:40.784111 18112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086 [term 1 LEADER]: Becoming Leader. State: Replica: 310bbad5f78b477fb5f7269797f74086, State: Running, Role: LEADER
I20260620 03:03:40.784708 18107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:03:40.784581 18112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310bbad5f78b477fb5f7269797f74086" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39649 } }
I20260620 03:03:40.787492 18114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 310bbad5f78b477fb5f7269797f74086. Latest consensus state: current_term: 1 leader_uuid: "310bbad5f78b477fb5f7269797f74086" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310bbad5f78b477fb5f7269797f74086" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39649 } } }
I20260620 03:03:40.787828 18114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086 [sys.catalog]: This master's current role is: LEADER
I20260620 03:03:40.788801 18113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "310bbad5f78b477fb5f7269797f74086" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310bbad5f78b477fb5f7269797f74086" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39649 } } }
I20260620 03:03:40.789908 18113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086 [sys.catalog]: This master's current role is: LEADER
I20260620 03:03:40.792583 18120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:03:40.798149 18120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:03:40.807019 18120 catalog_manager.cc:1347] Generated new cluster ID: a0ee7507aa7343ea865699ab597cb315
I20260620 03:03:40.807137 18120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:03:40.830998 18120 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:03:40.832368 18120 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:03:40.839843 18120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086: Generated new TSK 0
I20260620 03:03:40.840636 18120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:03:40.855486 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39649
--builtin_ntp_servers=127.0.97.20:37947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 03:03:41.209694 18131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:41.209964 18131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:41.210124 18131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:41.217991 18131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:41.218083 18131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:41.218269 18131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:03:41.227262 18131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:41.228545 18131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:41.230458 18131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:41.243247 18136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:41.243223 18137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:41.243801 18139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:41.244220 18131 server_base.cc:1034] running on GCE node
I20260620 03:03:41.244863 18131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:41.245925 18131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:41.247166 18131 hybrid_clock.cc:648] HybridClock initialized: now 1781924621247121 us; error 44 us; skew 500 ppm
I20260620 03:03:41.247498 18131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:03:41.249886 18131 webserver.cc:460] Webserver started at http://127.0.97.1:39363/ using document root <none> and password file <none>
I20260620 03:03:41.250617 18131 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:41.250752 18131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:41.251092 18131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:41.254144 18131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "48fa79a4ea9b4d8ebc452659caaafbc2"
format_stamp: "Formatted at 2026-06-20 03:03:41 on dist-test-slave-f1j0"
server_key: "bfd1b9d524b7b7dd49524a6c5697b725"
server_key_iv: "6a0a57fc8bf788969b94631f470e8f2e"
server_key_version: "encryptionkey@0"
I20260620 03:03:41.254917 18131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "48fa79a4ea9b4d8ebc452659caaafbc2"
format_stamp: "Formatted at 2026-06-20 03:03:41 on dist-test-slave-f1j0"
server_key: "bfd1b9d524b7b7dd49524a6c5697b725"
server_key_iv: "6a0a57fc8bf788969b94631f470e8f2e"
server_key_version: "encryptionkey@0"
I20260620 03:03:41.260062 18131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:03:41.263499 18145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:41.264853 18131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:03:41.265027 18131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "48fa79a4ea9b4d8ebc452659caaafbc2"
format_stamp: "Formatted at 2026-06-20 03:03:41 on dist-test-slave-f1j0"
server_key: "bfd1b9d524b7b7dd49524a6c5697b725"
server_key_iv: "6a0a57fc8bf788969b94631f470e8f2e"
server_key_version: "encryptionkey@0"
I20260620 03:03:41.265254 18131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:41.282459 18131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:41.283281 18131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:41.283573 18131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:41.284512 18131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:41.286135 18131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:41.286283 18131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:41.286403 18131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:41.286514 18131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:41.319585 18131 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41175
I20260620 03:03:41.319645 18257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41175 every 8 connection(s)
I20260620 03:03:41.320910 18131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:03:41.330751 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 18131
I20260620 03:03:41.330928 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:03:41.331403 388 external_mini_cluster.cc:1428] Setting key 95fb93ff0e9d9df7637860467cbd9d0f
I20260620 03:03:41.333462 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:41.323498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32951 (local address 127.0.97.62:39649)
0620 03:03:41.323940 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:41.323944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:41.324970 (+ 1026us) server_negotiation.cc:407] Connection header received
0620 03:03:41.326812 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:41.326817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:41.326907 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:41.327030 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:41.328167 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:41.328888 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:41.330521 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:41.331011 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:41.331153 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:41.332617 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:41.332622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:41.332626 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:41.332643 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:41.332687 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:41.332700 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:41.332803 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:41.333034 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 03:03:41.333134 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":229,"threads_started":1}
I20260620 03:03:41.334272 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39649
--builtin_ntp_servers=127.0.97.20:37947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 03:03:41.334501 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:41.323937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39649 (local address 127.0.97.1:32951)
0620 03:03:41.324815 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:41.324852 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:41.326540 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:41.327256 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:41.327264 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:41.327351 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:03:41.327995 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:41.328012 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:41.329064 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:41.329068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:41.330371 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:41.330383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:41.331242 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:41.331249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:41.331529 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:41.332283 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:03:41.332286 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:03:41.332326 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:03:41.332407 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:03:41.332801 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:03:41.332814 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:03:41.332825 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 03:03:41.333012 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 03:03:41.333408 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":263,"threads_started":1}
I20260620 03:03:41.336061 18258 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39649
I20260620 03:03:41.336359 18258 heartbeater.cc:464] Registering TS with master...
I20260620 03:03:41.337073 18258 heartbeater.cc:511] Master 127.0.97.62:39649 requested a full tablet report, sending...
I20260620 03:03:41.339397 18072 ts_manager.cc:194] Registered new tserver with Master: 48fa79a4ea9b4d8ebc452659caaafbc2 (127.0.97.1:41175)
I20260620 03:03:41.341614 18072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:32951
W20260620 03:03:41.659284 18262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:41.659572 18262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:41.659708 18262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:41.666970 18262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:41.667059 18262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:41.667246 18262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:03:41.676203 18262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:41.677495 18262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:41.679416 18262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:41.692493 18270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:41.692597 18268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:41.692795 18267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:41.693436 18262 server_base.cc:1034] running on GCE node
I20260620 03:03:41.694128 18262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:41.695056 18262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:41.696264 18262 hybrid_clock.cc:648] HybridClock initialized: now 1781924621696185 us; error 62 us; skew 500 ppm
I20260620 03:03:41.696610 18262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:03:41.698877 18262 webserver.cc:460] Webserver started at http://127.0.97.2:40829/ using document root <none> and password file <none>
I20260620 03:03:41.699566 18262 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:41.699679 18262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:41.700003 18262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:41.702901 18262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "0e1e92a0a2e149a9907ac3f9688eee9d"
format_stamp: "Formatted at 2026-06-20 03:03:41 on dist-test-slave-f1j0"
server_key: "a370c1849b5bf4d47131008ea009a0bc"
server_key_iv: "6b7b3e527c8663c5ad9d05af287dc0c4"
server_key_version: "encryptionkey@0"
I20260620 03:03:41.703684 18262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0e1e92a0a2e149a9907ac3f9688eee9d"
format_stamp: "Formatted at 2026-06-20 03:03:41 on dist-test-slave-f1j0"
server_key: "a370c1849b5bf4d47131008ea009a0bc"
server_key_iv: "6b7b3e527c8663c5ad9d05af287dc0c4"
server_key_version: "encryptionkey@0"
I20260620 03:03:41.708690 18262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 03:03:41.712049 18276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:41.713321 18262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 03:03:41.713500 18262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "0e1e92a0a2e149a9907ac3f9688eee9d"
format_stamp: "Formatted at 2026-06-20 03:03:41 on dist-test-slave-f1j0"
server_key: "a370c1849b5bf4d47131008ea009a0bc"
server_key_iv: "6b7b3e527c8663c5ad9d05af287dc0c4"
server_key_version: "encryptionkey@0"
I20260620 03:03:41.713743 18262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:41.728808 18262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:41.729697 18262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:41.729972 18262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:41.731017 18262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:41.732486 18262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:41.732607 18262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:41.732728 18262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:41.732817 18262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:41.766321 18262 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43445
I20260620 03:03:41.766364 18388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43445 every 8 connection(s)
I20260620 03:03:41.767655 18262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:03:41.770268 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 18262
I20260620 03:03:41.770464 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:03:41.771010 388 external_mini_cluster.cc:1428] Setting key 895aebaeb171defe5b1b2aa48a238a96
I20260620 03:03:41.773777 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39649
--builtin_ntp_servers=127.0.97.20:37947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 03:03:41.780570 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:41.770591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60255 (local address 127.0.97.62:39649)
0620 03:03:41.770794 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:41.770800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:41.772405 (+ 1605us) server_negotiation.cc:407] Connection header received
0620 03:03:41.774321 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:41.774326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:41.774416 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:41.774536 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:41.775590 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:41.776308 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:41.777906 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:41.778298 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:41.778434 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:41.779764 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:41.779769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:41.779772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:41.779790 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:41.779821 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:41.779833 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:41.779930 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:41.780160 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 03:03:41.780262 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 03:03:41.781561 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:41.771158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39649 (local address 127.0.97.2:60255)
0620 03:03:41.772235 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:41.772286 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:41.774062 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:41.774763 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:41.774771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:41.774861 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:03:41.775420 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:41.775435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:41.776459 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:41.776464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:41.777767 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:41.777779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:41.778519 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:41.778527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:41.778751 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:41.779463 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:03:41.779465 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:03:41.779504 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:03:41.779577 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:03:41.779928 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:03:41.779939 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:03:41.779949 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 03:03:41.780151 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 03:03:41.780524 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":344,"threads_started":1}
I20260620 03:03:41.782992 18389 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39649
I20260620 03:03:41.783313 18389 heartbeater.cc:464] Registering TS with master...
I20260620 03:03:41.784063 18389 heartbeater.cc:511] Master 127.0.97.62:39649 requested a full tablet report, sending...
I20260620 03:03:41.785660 18072 ts_manager.cc:194] Registered new tserver with Master: 0e1e92a0a2e149a9907ac3f9688eee9d (127.0.97.2:43445)
I20260620 03:03:41.786903 18072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:60255
W20260620 03:03:42.098433 18392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:42.098714 18392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:42.098799 18392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:42.105792 18392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:42.105886 18392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:42.105989 18392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:03:42.117987 18392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:42.119760 18392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:42.121496 18392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:42.134634 18400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:42.134752 18397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:42.134711 18398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:42.135720 18392 server_base.cc:1034] running on GCE node
I20260620 03:03:42.136401 18392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:42.137305 18392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:42.138499 18392 hybrid_clock.cc:648] HybridClock initialized: now 1781924622138430 us; error 65 us; skew 500 ppm
I20260620 03:03:42.138804 18392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:03:42.141013 18392 webserver.cc:460] Webserver started at http://127.0.97.3:35589/ using document root <none> and password file <none>
I20260620 03:03:42.141748 18392 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:42.141860 18392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:42.142184 18392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:42.144893 18392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "b9c4f10000064e3cb339ab86192206c4"
format_stamp: "Formatted at 2026-06-20 03:03:42 on dist-test-slave-f1j0"
server_key: "e7c3e587ab8fd6808f1ccac56b8bc68a"
server_key_iv: "a3ef8a739abd601a2749f678431dcdf1"
server_key_version: "encryptionkey@0"
I20260620 03:03:42.145723 18392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b9c4f10000064e3cb339ab86192206c4"
format_stamp: "Formatted at 2026-06-20 03:03:42 on dist-test-slave-f1j0"
server_key: "e7c3e587ab8fd6808f1ccac56b8bc68a"
server_key_iv: "a3ef8a739abd601a2749f678431dcdf1"
server_key_version: "encryptionkey@0"
I20260620 03:03:42.150890 18392 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 03:03:42.154222 18406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:42.155526 18392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:03:42.155651 18392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "b9c4f10000064e3cb339ab86192206c4"
format_stamp: "Formatted at 2026-06-20 03:03:42 on dist-test-slave-f1j0"
server_key: "e7c3e587ab8fd6808f1ccac56b8bc68a"
server_key_iv: "a3ef8a739abd601a2749f678431dcdf1"
server_key_version: "encryptionkey@0"
I20260620 03:03:42.155836 18392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:42.182220 18392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:42.183022 18392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:42.183259 18392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:42.184175 18392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:42.185707 18392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:42.185793 18392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:42.185875 18392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:42.185930 18392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:42.219827 18392 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42915
I20260620 03:03:42.219915 18518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42915 every 8 connection(s)
I20260620 03:03:42.221148 18392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:03:42.228942 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 18392
I20260620 03:03:42.229125 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:03:42.229799 388 external_mini_cluster.cc:1428] Setting key cde9cfad81a5fcaaa536e0ef41a1eca0
I20260620 03:03:42.234058 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.223860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41545 (local address 127.0.97.62:39649)
0620 03:03:42.224088 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:42.224094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:42.225448 (+ 1354us) server_negotiation.cc:407] Connection header received
0620 03:03:42.227163 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:42.227168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:42.227235 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:42.227323 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:42.228319 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.229402 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.231140 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.231756 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.231863 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:42.233348 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:42.233352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:42.233355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:42.233373 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:42.233402 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:42.233413 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:42.233481 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:42.233746 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 03:03:42.233820 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 03:03:42.235069 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.224416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39649 (local address 127.0.97.3:41545)
0620 03:03:42.225303 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:42.225339 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:42.226922 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:42.227523 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:42.227530 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:42.227612 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:03:42.228166 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:42.228186 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.229547 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.229550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:42.231013 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:42.231023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.231994 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.232001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:42.232287 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:42.233058 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:03:42.233060 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:03:42.233103 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:03:42.233177 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:03:42.233497 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:03:42.233504 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:03:42.233509 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 03:03:42.233723 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 03:03:42.234015 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":273,"threads_started":1}
I20260620 03:03:42.236410 18519 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39649
I20260620 03:03:42.236663 18519 heartbeater.cc:464] Registering TS with master...
I20260620 03:03:42.237313 18519 heartbeater.cc:511] Master 127.0.97.62:39649 requested a full tablet report, sending...
I20260620 03:03:42.238627 18072 ts_manager.cc:194] Registered new tserver with Master: b9c4f10000064e3cb339ab86192206c4 (127.0.97.3:42915)
I20260620 03:03:42.239576 18072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:41545
I20260620 03:03:42.244726 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:03:42.254637 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.248512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39608 (local address 127.0.97.62:39649)
0620 03:03:42.248694 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:42.248703 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:42.249111 (+ 408us) server_negotiation.cc:407] Connection header received
0620 03:03:42.249303 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:42.249308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:42.249391 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:42.249493 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:42.250059 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.250749 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.252222 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.252633 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.252758 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:42.253905 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:42.253911 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:42.253916 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:42.253942 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:42.253972 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:42.253984 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:42.254082 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:42.254231 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 03:03:42.254322 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 03:03:42.258564 18072 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:03:42.265646 18072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:03:42.300586 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.278047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39256 (local address 127.0.97.2:43445)
0620 03:03:42.289369 (+ 11322us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:42.289379 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:42.290078 (+ 699us) server_negotiation.cc:407] Connection header received
0620 03:03:42.290191 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:42.290201 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:42.290461 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:42.291475 (+ 1014us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:42.291721 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.293783 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.297452 (+ 3669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.299058 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.299178 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:42.299426 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:42.300064 (+ 638us) server_negotiation.cc:299] Negotiation successful
0620 03:03:42.300227 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11163,"thread_start_us":11065,"threads_started":1}
I20260620 03:03:42.300621 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.277386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43445 (local address 127.0.0.1:39256)
0620 03:03:42.289790 (+ 12404us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:42.289807 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:42.289930 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:42.290706 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:42.290714 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:42.290744 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:42.291197 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:42.291212 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.293984 (+ 2772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.293993 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:42.297164 (+ 3171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:42.297184 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.299606 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.299614 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:42.299800 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:42.299865 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 03:03:42.300001 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 03:03:42.300207 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12261,"mutex_wait_us":25,"thread_start_us":12098,"threads_started":1}
I20260620 03:03:42.303259 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.277089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41175 (local address 127.0.0.1:46920)
0620 03:03:42.289105 (+ 12016us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:42.289162 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:42.289421 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:42.294495 (+ 5074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:42.294498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:42.294518 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:42.295122 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:42.295140 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.298490 (+ 3350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.298495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:42.300442 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:42.300455 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.302375 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.302382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:42.302598 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:42.302633 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 03:03:42.302734 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 03:03:42.302870 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11665,"mutex_wait_us":172,"thread_start_us":12594,"threads_started":1}
I20260620 03:03:42.303300 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.277265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.0.97.1:41175)
0620 03:03:42.293538 (+ 16273us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:42.293548 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:42.293597 (+ 49us) server_negotiation.cc:407] Connection header received
0620 03:03:42.293692 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:42.293699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:42.293942 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:42.294146 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:42.295409 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.298369 (+ 2960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.300674 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.302159 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.302297 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:42.302544 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:42.302798 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 03:03:42.302972 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16082,"thread_start_us":17050,"threads_started":1}
I20260620 03:03:42.305080 18324 tablet_service.cc:1467] Processing CreateTablet for tablet 12798c777d36486bb631e87c619610ab (DEFAULT_TABLE table=test-table [id=5b3477931b404e9eb4674f6b9c0ad85e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:03:42.306341 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.294810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42915 (local address 127.0.0.1:51776)
0620 03:03:42.298385 (+ 3575us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:42.298402 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:42.298527 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:42.299194 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:42.299199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:42.299223 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:42.299738 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:42.299753 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.301965 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.301969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:42.303974 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:42.303985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.305551 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.305557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:42.305765 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:42.305800 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 03:03:42.305900 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 03:03:42.306040 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3461,"thread_start_us":3361,"threads_started":1}
I20260620 03:03:42.306483 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.290717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.0.97.3:42915)
0620 03:03:42.298153 (+ 7436us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:42.298161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:42.298657 (+ 496us) server_negotiation.cc:407] Connection header received
0620 03:03:42.298759 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:42.298767 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:42.298987 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:42.299186 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:42.299940 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.301842 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.304144 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.305314 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.305464 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:42.305767 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:42.305959 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 03:03:42.306131 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7274,"thread_start_us":7128,"threads_started":1}
I20260620 03:03:42.306952 18324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12798c777d36486bb631e87c619610ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:42.307467 18193 tablet_service.cc:1467] Processing CreateTablet for tablet 12798c777d36486bb631e87c619610ab (DEFAULT_TABLE table=test-table [id=5b3477931b404e9eb4674f6b9c0ad85e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:03:42.308584 18193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12798c777d36486bb631e87c619610ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:42.317984 18537 tablet_bootstrap.cc:492] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d: Bootstrap starting.
I20260620 03:03:42.317768 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 12798c777d36486bb631e87c619610ab (DEFAULT_TABLE table=test-table [id=5b3477931b404e9eb4674f6b9c0ad85e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:03:42.319310 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12798c777d36486bb631e87c619610ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:42.323274 18537 tablet_bootstrap.cc:654] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:42.324879 18537 log.cc:824] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:42.332281 18538 tablet_bootstrap.cc:492] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2: Bootstrap starting.
I20260620 03:03:42.333910 18540 tablet_bootstrap.cc:492] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4: Bootstrap starting.
I20260620 03:03:42.335310 18537 tablet_bootstrap.cc:492] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d: No bootstrap required, opened a new log
I20260620 03:03:42.335610 18537 ts_tablet_manager.cc:1397] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.007s
I20260620 03:03:42.336916 18538 tablet_bootstrap.cc:654] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:42.337363 18540 tablet_bootstrap.cc:654] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:42.338277 18538 log.cc:824] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:42.338630 18540 log.cc:824] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:42.341392 18540 tablet_bootstrap.cc:492] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4: No bootstrap required, opened a new log
I20260620 03:03:42.341696 18540 ts_tablet_manager.cc:1397] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260620 03:03:42.341867 18538 tablet_bootstrap.cc:492] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2: No bootstrap required, opened a new log
I20260620 03:03:42.342110 18538 ts_tablet_manager.cc:1397] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 03:03:42.344694 18258 heartbeater.cc:503] Master 127.0.97.62:39649 was elected leader, sending a full tablet report...
I20260620 03:03:42.345314 18537 raft_consensus.cc:348] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48fa79a4ea9b4d8ebc452659caaafbc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41175 } } peers { permanent_uuid: "0e1e92a0a2e149a9907ac3f9688eee9d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43445 } } peers { permanent_uuid: "b9c4f10000064e3cb339ab86192206c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42915 } }
I20260620 03:03:42.345763 18537 raft_consensus.cc:374] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:42.345855 18537 raft_consensus.cc:729] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1e92a0a2e149a9907ac3f9688eee9d, State: Initialized, Role: FOLLOWER
I20260620 03:03:42.346567 18537 consensus_queue.cc:260] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48fa79a4ea9b4d8ebc452659caaafbc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41175 } } peers { permanent_uuid: "0e1e92a0a2e149a9907ac3f9688eee9d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43445 } } peers { permanent_uuid: "b9c4f10000064e3cb339ab86192206c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42915 } }
I20260620 03:03:42.350113 18389 heartbeater.cc:503] Master 127.0.97.62:39649 was elected leader, sending a full tablet report...
I20260620 03:03:42.351142 18537 ts_tablet_manager.cc:1428] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260620 03:03:42.351325 18540 raft_consensus.cc:348] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48fa79a4ea9b4d8ebc452659caaafbc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41175 } } peers { permanent_uuid: "0e1e92a0a2e149a9907ac3f9688eee9d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43445 } } peers { permanent_uuid: "b9c4f10000064e3cb339ab86192206c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42915 } }
I20260620 03:03:42.351801 18540 raft_consensus.cc:374] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:42.351929 18540 raft_consensus.cc:729] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9c4f10000064e3cb339ab86192206c4, State: Initialized, Role: FOLLOWER
I20260620 03:03:42.352573 18540 consensus_queue.cc:260] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48fa79a4ea9b4d8ebc452659caaafbc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41175 } } peers { permanent_uuid: "0e1e92a0a2e149a9907ac3f9688eee9d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43445 } } peers { permanent_uuid: "b9c4f10000064e3cb339ab86192206c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42915 } }
I20260620 03:03:42.353585 18538 raft_consensus.cc:348] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48fa79a4ea9b4d8ebc452659caaafbc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41175 } } peers { permanent_uuid: "0e1e92a0a2e149a9907ac3f9688eee9d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43445 } } peers { permanent_uuid: "b9c4f10000064e3cb339ab86192206c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42915 } }
I20260620 03:03:42.354082 18538 raft_consensus.cc:374] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:42.354120 18519 heartbeater.cc:503] Master 127.0.97.62:39649 was elected leader, sending a full tablet report...
I20260620 03:03:42.354225 18538 raft_consensus.cc:729] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48fa79a4ea9b4d8ebc452659caaafbc2, State: Initialized, Role: FOLLOWER
I20260620 03:03:42.354930 18540 ts_tablet_manager.cc:1428] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260620 03:03:42.354794 18538 consensus_queue.cc:260] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48fa79a4ea9b4d8ebc452659caaafbc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41175 } } peers { permanent_uuid: "0e1e92a0a2e149a9907ac3f9688eee9d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43445 } } peers { permanent_uuid: "b9c4f10000064e3cb339ab86192206c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42915 } }
I20260620 03:03:42.356956 18538 ts_tablet_manager.cc:1428] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260620 03:03:42.424177 18545 raft_consensus.cc:482] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:03:42.424589 18545 raft_consensus.cc:504] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48fa79a4ea9b4d8ebc452659caaafbc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41175 } } peers { permanent_uuid: "0e1e92a0a2e149a9907ac3f9688eee9d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43445 } } peers { permanent_uuid: "b9c4f10000064e3cb339ab86192206c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42915 } }
I20260620 03:03:42.426760 18545 leader_election.cc:288] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e1e92a0a2e149a9907ac3f9688eee9d (127.0.97.2:43445), b9c4f10000064e3cb339ab86192206c4 (127.0.97.3:42915)
I20260620 03:03:42.433570 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.427139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43445 (local address 127.0.97.1:58647)
0620 03:03:42.427865 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:42.427887 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:42.428047 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:42.428457 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:42.428463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:42.428495 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:42.428829 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:42.428837 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.429886 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.429890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:42.431312 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:42.431323 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.432988 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.432997 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:42.433138 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:42.433162 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 03:03:42.433236 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 03:03:42.433320 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":402,"threads_started":1}
I20260620 03:03:42.433727 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.427261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58647 (local address 127.0.97.2:43445)
0620 03:03:42.427455 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:42.427459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:42.427979 (+ 520us) server_negotiation.cc:407] Connection header received
0620 03:03:42.428198 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:42.428204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:42.428303 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:42.428406 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:42.428988 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.429760 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.431499 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.432592 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.432735 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:42.432852 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:42.433299 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 03:03:42.433435 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 03:03:42.434499 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.428168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59313 (local address 127.0.97.3:42915)
0620 03:03:42.428350 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:42.428358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:42.428741 (+ 383us) server_negotiation.cc:407] Connection header received
0620 03:03:42.428944 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:42.428948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:42.429007 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:42.429106 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:42.429643 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.430360 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.432089 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.433474 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.433622 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:42.433712 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:42.434149 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 03:03:42.434271 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 03:03:42.434536 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.428007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42915 (local address 127.0.97.1:59313)
0620 03:03:42.428623 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:42.428639 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:42.428790 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:42.429119 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:42.429124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:42.429146 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:42.429477 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:42.429486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.430504 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.430516 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:42.431849 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:42.431861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.433687 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.433695 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:42.433880 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:42.434007 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 03:03:42.434122 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 03:03:42.434242 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":330,"threads_started":1}
I20260620 03:03:42.434782 18344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12798c777d36486bb631e87c619610ab" candidate_uuid: "48fa79a4ea9b4d8ebc452659caaafbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1e92a0a2e149a9907ac3f9688eee9d" is_pre_election: true
I20260620 03:03:42.435388 18344 raft_consensus.cc:2454] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48fa79a4ea9b4d8ebc452659caaafbc2 in term 0.
I20260620 03:03:42.435541 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12798c777d36486bb631e87c619610ab" candidate_uuid: "48fa79a4ea9b4d8ebc452659caaafbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9c4f10000064e3cb339ab86192206c4" is_pre_election: true
I20260620 03:03:42.436065 18474 raft_consensus.cc:2454] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48fa79a4ea9b4d8ebc452659caaafbc2 in term 0.
I20260620 03:03:42.436156 18148 leader_election.cc:302] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1e92a0a2e149a9907ac3f9688eee9d, 48fa79a4ea9b4d8ebc452659caaafbc2; no voters:
I20260620 03:03:42.436688 18545 raft_consensus.cc:2781] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:03:42.436820 18545 raft_consensus.cc:482] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:03:42.436959 18545 raft_consensus.cc:3037] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:42.439199 18545 raft_consensus.cc:504] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48fa79a4ea9b4d8ebc452659caaafbc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41175 } } peers { permanent_uuid: "0e1e92a0a2e149a9907ac3f9688eee9d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43445 } } peers { permanent_uuid: "b9c4f10000064e3cb339ab86192206c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42915 } }
I20260620 03:03:42.439872 18545 leader_election.cc:288] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [CANDIDATE]: Term 1 election: Requested vote from peers 0e1e92a0a2e149a9907ac3f9688eee9d (127.0.97.2:43445), b9c4f10000064e3cb339ab86192206c4 (127.0.97.3:42915)
I20260620 03:03:42.440306 18344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12798c777d36486bb631e87c619610ab" candidate_uuid: "48fa79a4ea9b4d8ebc452659caaafbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1e92a0a2e149a9907ac3f9688eee9d"
I20260620 03:03:42.440414 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12798c777d36486bb631e87c619610ab" candidate_uuid: "48fa79a4ea9b4d8ebc452659caaafbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9c4f10000064e3cb339ab86192206c4"
I20260620 03:03:42.440511 18344 raft_consensus.cc:3037] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:42.440657 18474 raft_consensus.cc:3037] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:42.442824 18344 raft_consensus.cc:2454] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48fa79a4ea9b4d8ebc452659caaafbc2 in term 1.
I20260620 03:03:42.442920 18474 raft_consensus.cc:2454] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48fa79a4ea9b4d8ebc452659caaafbc2 in term 1.
I20260620 03:03:42.443243 18148 leader_election.cc:302] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1e92a0a2e149a9907ac3f9688eee9d, 48fa79a4ea9b4d8ebc452659caaafbc2; no voters:
I20260620 03:03:42.443506 18545 raft_consensus.cc:2781] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:03:42.443925 18545 raft_consensus.cc:686] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [term 1 LEADER]: Becoming Leader. State: Replica: 48fa79a4ea9b4d8ebc452659caaafbc2, State: Running, Role: LEADER
I20260620 03:03:42.444427 18545 consensus_queue.cc:237] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48fa79a4ea9b4d8ebc452659caaafbc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41175 } } peers { permanent_uuid: "0e1e92a0a2e149a9907ac3f9688eee9d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43445 } } peers { permanent_uuid: "b9c4f10000064e3cb339ab86192206c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42915 } }
I20260620 03:03:42.449836 18072 catalog_manager.cc:5495] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48fa79a4ea9b4d8ebc452659caaafbc2 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "48fa79a4ea9b4d8ebc452659caaafbc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48fa79a4ea9b4d8ebc452659caaafbc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e1e92a0a2e149a9907ac3f9688eee9d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9c4f10000064e3cb339ab86192206c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42915 } health_report { overall_health: UNKNOWN } } }
I20260620 03:03:42.467747 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.462650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46922 (local address 127.0.97.1:41175)
0620 03:03:42.462847 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:42.462852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:42.462871 (+ 19us) server_negotiation.cc:407] Connection header received
0620 03:03:42.463078 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:42.463084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:42.463218 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:42.463369 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:03:42.463708 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.464395 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.465778 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.466202 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.466293 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:42.466728 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:03:42.467214 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:03:42.467320 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:42.467385 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 03:03:42.467444 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 03:03:42.472766 18520 tablet.cc:2369] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:03:42.475931 18474 raft_consensus.cc:1261] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4 [term 1 FOLLOWER]: Refusing update from remote peer 48fa79a4ea9b4d8ebc452659caaafbc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:03:42.475930 18344 raft_consensus.cc:1261] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d [term 1 FOLLOWER]: Refusing update from remote peer 48fa79a4ea9b4d8ebc452659caaafbc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:03:42.476881 18548 consensus_queue.cc:1035] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9c4f10000064e3cb339ab86192206c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:03:42.477253 18545 consensus_queue.cc:1035] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e1e92a0a2e149a9907ac3f9688eee9d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:03:42.504694 18072 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:03:42.504956 18072 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:03:42.508556 18072 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 310bbad5f78b477fb5f7269797f74086: Sending DeleteTablet for 3 replicas of tablet 12798c777d36486bb631e87c619610ab
I20260620 03:03:42.509757 18193 tablet_service.cc:1514] Processing DeleteTablet for tablet 12798c777d36486bb631e87c619610ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:03:42 UTC) from {username='slave'} at 127.0.0.1:46920
I20260620 03:03:42.510128 18324 tablet_service.cc:1514] Processing DeleteTablet for tablet 12798c777d36486bb631e87c619610ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:03:42 UTC) from {username='slave'} at 127.0.0.1:39256
I20260620 03:03:42.510792 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 12798c777d36486bb631e87c619610ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:03:42 UTC) from {username='slave'} at 127.0.0.1:51776
I20260620 03:03:42.511030 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 18131
I20260620 03:03:42.524834 18563 tablet_replica.cc:330] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4: stopping tablet replica
I20260620 03:03:42.525496 18563 raft_consensus.cc:2229] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:03:42.526183 18563 raft_consensus.cc:2258] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:03:42.528774 18563 ts_tablet_manager.cc:1905] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:03:42.532317 18561 tablet_replica.cc:330] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2: stopping tablet replica
I20260620 03:03:42.532603 18562 tablet_replica.cc:330] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d: stopping tablet replica
I20260620 03:03:42.532804 18561 raft_consensus.cc:2229] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:03:42.532994 18562 raft_consensus.cc:2229] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:03:42.533500 18561 raft_consensus.cc:2258] T 12798c777d36486bb631e87c619610ab P 48fa79a4ea9b4d8ebc452659caaafbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:03:42.533525 18562 raft_consensus.cc:2258] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:03:42.535723 18562 ts_tablet_manager.cc:1905] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:03:42.536034 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.513687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46932 (local address 127.0.97.1:41175)
0620 03:03:42.514155 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:42.514163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:42.522657 (+ 8494us) server_negotiation.cc:407] Connection header received
0620 03:03:42.523200 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:42.523209 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:42.523355 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:42.523458 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:42.526742 (+ 3284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.528000 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.530158 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.530834 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.530935 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:42.534254 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:42.534264 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:42.534281 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:42.535091 (+ 810us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:42.535151 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:42.535181 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:42.535282 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:42.535550 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 03:03:42.535657 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315}
I20260620 03:03:42.537356 18253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:42.539004 18563 ts_tablet_manager.cc:1918] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:03:42.539209 18563 log.cc:1196] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/12798c777d36486bb631e87c619610ab
I20260620 03:03:42.539887 18563 ts_tablet_manager.cc:1939] T 12798c777d36486bb631e87c619610ab P b9c4f10000064e3cb339ab86192206c4: Deleting consensus metadata
I20260620 03:03:42.542619 18058 catalog_manager.cc:4841] TS b9c4f10000064e3cb339ab86192206c4 (127.0.97.3:42915): tablet 12798c777d36486bb631e87c619610ab (table test-table [id=5b3477931b404e9eb4674f6b9c0ad85e]) successfully deleted
I20260620 03:03:42.544006 18562 ts_tablet_manager.cc:1918] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:03:42.544163 18562 log.cc:1196] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/12798c777d36486bb631e87c619610ab
I20260620 03:03:42.544644 18562 ts_tablet_manager.cc:1939] T 12798c777d36486bb631e87c619610ab P 0e1e92a0a2e149a9907ac3f9688eee9d: Deleting consensus metadata
I20260620 03:03:42.546135 18058 catalog_manager.cc:4841] TS 0e1e92a0a2e149a9907ac3f9688eee9d (127.0.97.2:43445): tablet 12798c777d36486bb631e87c619610ab (table test-table [id=5b3477931b404e9eb4674f6b9c0ad85e]) successfully deleted
I20260620 03:03:42.634094 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 18131
I20260620 03:03:42.652709 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 18262
W20260620 03:03:42.652712 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:41175 (error 108)
I20260620 03:03:42.654731 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.654295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41175 (local address 127.0.0.1:46944)
0620 03:03:42.654476 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:42.654596 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 03:03:42.654978 18056 catalog_manager.cc:4584] TS 48fa79a4ea9b4d8ebc452659caaafbc2 (127.0.97.1:41175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12798c777d36486bb631e87c619610ab: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41175: connect: Connection refused (error 111)
I20260620 03:03:42.659911 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.653880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39262 (local address 127.0.97.2:43445)
0620 03:03:42.654106 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:42.654113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:42.654135 (+ 22us) server_negotiation.cc:407] Connection header received
0620 03:03:42.654205 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:42.654208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:42.654292 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:42.654409 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:42.654800 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.655678 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.656959 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.657356 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.657481 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:42.658520 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:42.658528 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:42.658542 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:42.659184 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:42.659228 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:42.659248 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:42.659365 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:42.659496 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 03:03:42.659608 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 03:03:42.660619 18384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:42.708765 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.708189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41175 (local address 127.0.0.1:46956)
0620 03:03:42.708450 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:42.708566 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 03:03:42.709885 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.709443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41175 (local address 127.0.0.1:46964)
0620 03:03:42.709643 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:42.709729 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 03:03:42.750213 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 18262
I20260620 03:03:42.770421 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.769905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41175 (local address 127.0.0.1:46970)
0620 03:03:42.770112 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:42.770255 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 03:03:42.771575 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.771158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41175 (local address 127.0.0.1:46972)
0620 03:03:42.771342 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:42.771451 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 03:03:42.773417 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 18392
I20260620 03:03:42.780949 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.774269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.0.97.3:42915)
0620 03:03:42.774447 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:42.774451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:42.774475 (+ 24us) server_negotiation.cc:407] Connection header received
0620 03:03:42.774666 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:42.774670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:42.774764 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:42.774845 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:42.775311 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.776060 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.777948 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:42.778356 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:42.778451 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:42.779590 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:42.779599 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:42.779615 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:42.780206 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:42.780254 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:42.780273 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:42.780368 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:42.780494 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 03:03:42.780586 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 03:03:42.781653 18514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:42.852356 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.851788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41175 (local address 127.0.0.1:46984)
0620 03:03:42.852044 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:42.852169 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 03:03:42.853458 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:42.853030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41175 (local address 127.0.0.1:46986)
0620 03:03:42.853234 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:42.853334 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 03:03:42.870518 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 18392
I20260620 03:03:42.887306 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 18041
I20260620 03:03:42.888374 18102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:42.997121 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 18041
2026-06-20T03:03:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2767 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:03:43 dist-test-slave-f1j0 krb5kdc[18572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:03:43 dist-test-slave-f1j0 krb5kdc[18572](info): set up 2 sockets
Jun 20 03:03:43 dist-test-slave-f1j0 krb5kdc[18572](info): commencing operation
krb5kdc: starting...
W20260620 03:03:44.105801 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:03:44 dist-test-slave-f1j0 krb5kdc[18572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:03:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:03:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:03:44.350677 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38623
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:03:44.681113 18585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:44.681397 18585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:44.681602 18585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:44.688814 18585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:03:44.688930 18585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:44.689034 18585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:44.689152 18585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:03:44.689218 18585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:03:44.699577 18585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:44.700966 18585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:44.702353 18585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:44.714650 18590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:44.714740 18591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:44.714830 18593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:44.715675 18585 server_base.cc:1034] running on GCE node
I20260620 03:03:44.716387 18585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:44.717798 18585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:44.718998 18585 hybrid_clock.cc:648] HybridClock initialized: now 1781924624718960 us; error 34 us; skew 500 ppm
I20260620 03:03:44.719357 18585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:03:44 dist-test-slave-f1j0 krb5kdc[18572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:03:44.722584 18585 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:03:44.724120 18585 webserver.cc:460] Webserver started at http://127.0.97.62:45533/ using document root <none> and password file <none>
I20260620 03:03:44.724833 18585 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:44.724980 18585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:44.725294 18585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:44.728328 18585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "2543967755574e85a2f88924b22e7231"
format_stamp: "Formatted at 2026-06-20 03:03:44 on dist-test-slave-f1j0"
server_key: "6e66416afb1e071c1b1c2014747f6397"
server_key_iv: "4021b784b235c2a727f6045f90bb5af0"
server_key_version: "encryptionkey@0"
I20260620 03:03:44.729102 18585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "2543967755574e85a2f88924b22e7231"
format_stamp: "Formatted at 2026-06-20 03:03:44 on dist-test-slave-f1j0"
server_key: "6e66416afb1e071c1b1c2014747f6397"
server_key_iv: "4021b784b235c2a727f6045f90bb5af0"
server_key_version: "encryptionkey@0"
I20260620 03:03:44.734241 18585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 03:03:44.737695 18600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:44.739116 18585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:03:44.739300 18585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "2543967755574e85a2f88924b22e7231"
format_stamp: "Formatted at 2026-06-20 03:03:44 on dist-test-slave-f1j0"
server_key: "6e66416afb1e071c1b1c2014747f6397"
server_key_iv: "4021b784b235c2a727f6045f90bb5af0"
server_key_version: "encryptionkey@0"
I20260620 03:03:44.739511 18585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:44.754743 18585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:44.759256 18585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:44.759560 18585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:44.778191 18585 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38623
I20260620 03:03:44.778226 18651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38623 every 8 connection(s)
I20260620 03:03:44.779618 18585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:03:44.783668 18652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:44.786722 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 18585
I20260620 03:03:44.786918 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:03:44.787456 388 external_mini_cluster.cc:1428] Setting key 444c6b40d1342d3631360a3e5e5549bd
I20260620 03:03:44.796094 18652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231: Bootstrap starting.
I20260620 03:03:44.800086 18652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:44.801496 18652 log.cc:824] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231: Log is configured to *not* fsync() on all Append() calls
Jun 20 03:03:44 dist-test-slave-f1j0 krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:03:44.804868 18652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231: No bootstrap required, opened a new log
I20260620 03:03:44.811611 18655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:44.789391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54810 (local address 127.0.97.62:38623)
0620 03:03:44.790220 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:44.790232 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:44.790273 (+ 41us) server_negotiation.cc:407] Connection header received
0620 03:03:44.791914 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:44.791935 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:44.792341 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:44.792692 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:44.794503 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:44.795763 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:44.797697 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:44.798482 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:44.798615 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:44.803982 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:44.803989 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:44.804005 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:44.804055 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:44.807327 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:44.808207 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:44.808212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:44.808219 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:44.808397 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:44.808808 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:44.808811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:44.808814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:44.809339 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:44.809679 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:44.810095 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 03:03:44.810519 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":405,"threads_started":1}
I20260620 03:03:44.813872 18652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2543967755574e85a2f88924b22e7231" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38623 } }
I20260620 03:03:44.814270 18652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:44.814404 18652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2543967755574e85a2f88924b22e7231, State: Initialized, Role: FOLLOWER
I20260620 03:03:44.814970 18652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2543967755574e85a2f88924b22e7231" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38623 } }
I20260620 03:03:44.815223 18652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:03:44.815369 18652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:03:44.815537 18652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:44.817775 18652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2543967755574e85a2f88924b22e7231" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38623 } }
I20260620 03:03:44.818262 18652 leader_election.cc:302] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2543967755574e85a2f88924b22e7231; no voters:
I20260620 03:03:44.818867 18652 leader_election.cc:288] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:03:44.819000 18657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:03:44.819692 18657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231 [term 1 LEADER]: Becoming Leader. State: Replica: 2543967755574e85a2f88924b22e7231, State: Running, Role: LEADER
I20260620 03:03:44.820286 18657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2543967755574e85a2f88924b22e7231" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38623 } }
I20260620 03:03:44.821097 18652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:03:44.823493 18659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2543967755574e85a2f88924b22e7231. Latest consensus state: current_term: 1 leader_uuid: "2543967755574e85a2f88924b22e7231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2543967755574e85a2f88924b22e7231" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38623 } } }
I20260620 03:03:44.823472 18658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2543967755574e85a2f88924b22e7231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2543967755574e85a2f88924b22e7231" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38623 } } }
I20260620 03:03:44.823927 18658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231 [sys.catalog]: This master's current role is: LEADER
I20260620 03:03:44.824496 18659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231 [sys.catalog]: This master's current role is: LEADER
I20260620 03:03:44.825773 18664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:03:44.830402 18664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:03:44.838812 18664 catalog_manager.cc:1347] Generated new cluster ID: 31833802fe794d23a393c5263e0032e7
I20260620 03:03:44.838934 18664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:03:44.862076 18664 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:03:44.863511 18664 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:03:44.876242 18664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231: Generated new TSK 0
I20260620 03:03:44.877149 18664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:03:44.948222 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38623
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:03:45.277454 18678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:45.277777 18678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:45.277874 18678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:45.285077 18678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:45.285189 18678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:45.285311 18678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:03:45.294955 18678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:45.296206 18678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:45.297439 18678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:45.310784 18684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:45.310989 18683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:45.311432 18686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:45.311973 18678 server_base.cc:1034] running on GCE node
I20260620 03:03:45.312668 18678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:45.313722 18678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:45.314970 18678 hybrid_clock.cc:648] HybridClock initialized: now 1781924625314922 us; error 56 us; skew 500 ppm
I20260620 03:03:45.315299 18678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:03:45 dist-test-slave-f1j0 krb5kdc[18572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:03:45.318506 18678 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:03:45.320003 18678 webserver.cc:460] Webserver started at http://127.0.97.1:44063/ using document root <none> and password file <none>
I20260620 03:03:45.320685 18678 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:45.320782 18678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:45.321136 18678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:45.324127 18678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "e2ffe153162547e9b98c7eee0f1e8a4a"
format_stamp: "Formatted at 2026-06-20 03:03:45 on dist-test-slave-f1j0"
server_key: "80ad939a7926331b154dbd604d8a336a"
server_key_iv: "d5e94afe6d6275bd7108f3f5a3a3a66e"
server_key_version: "encryptionkey@0"
I20260620 03:03:45.324957 18678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e2ffe153162547e9b98c7eee0f1e8a4a"
format_stamp: "Formatted at 2026-06-20 03:03:45 on dist-test-slave-f1j0"
server_key: "80ad939a7926331b154dbd604d8a336a"
server_key_iv: "d5e94afe6d6275bd7108f3f5a3a3a66e"
server_key_version: "encryptionkey@0"
I20260620 03:03:45.330256 18678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 03:03:45.333674 18693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:45.334990 18678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:03:45.335122 18678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "e2ffe153162547e9b98c7eee0f1e8a4a"
format_stamp: "Formatted at 2026-06-20 03:03:45 on dist-test-slave-f1j0"
server_key: "80ad939a7926331b154dbd604d8a336a"
server_key_iv: "d5e94afe6d6275bd7108f3f5a3a3a66e"
server_key_version: "encryptionkey@0"
I20260620 03:03:45.335287 18678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:45.363245 18678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:45.367321 18678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:45.367583 18678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:45.368489 18678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:45.369851 18678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:45.369928 18678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:45.370015 18678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:45.370060 18678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:45.403365 18678 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41033
I20260620 03:03:45.403420 18805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41033 every 8 connection(s)
I20260620 03:03:45.404714 18678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:03:45.411765 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 18678
I20260620 03:03:45.411950 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:03:45.412402 388 external_mini_cluster.cc:1428] Setting key aa87b9b0530c19313f67974a67a01940
Jun 20 03:03:45 dist-test-slave-f1j0 krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:03:45.426640 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:45.407237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51101 (local address 127.0.97.62:38623)
0620 03:03:45.407702 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:45.407709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:45.408620 (+ 911us) server_negotiation.cc:407] Connection header received
0620 03:03:45.410595 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:45.410601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:45.410685 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:45.410788 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:45.412457 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:45.413393 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:45.415114 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:45.415868 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:45.416014 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:45.419409 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:45.419418 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:45.419426 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:45.419471 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:45.422872 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:45.423828 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:45.423831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:45.423834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:45.423948 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:45.424428 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:45.424431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:45.424433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:45.424683 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:45.424791 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:45.425968 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0620 03:03:45.426300 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":223,"threads_started":1}
I20260620 03:03:45.427773 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:45.407793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38623 (local address 127.0.97.1:51101)
0620 03:03:45.408464 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:45.408502 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:45.410271 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:45.410992 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:45.410999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:45.411623 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:03:45.412279 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:45.412297 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:45.413635 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:45.413639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:45.414967 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:45.414983 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:45.416093 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:45.416098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:45.416295 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:45.416971 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:03:45.416993 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:03:45.419194 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:03:45.423006 (+ 3812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:03:45.423018 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:03:45.423044 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:03:45.423704 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:03:45.424059 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:03:45.424066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:03:45.424073 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:03:45.424317 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:03:45.424807 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:03:45.424818 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:03:45.425315 (+ 497us) client_negotiation.cc:769] Sending connection context
0620 03:03:45.425645 (+ 330us) client_negotiation.cc:240] Negotiation successful
0620 03:03:45.426264 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":273,"threads_started":1}
I20260620 03:03:45.429529 18806 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38623
I20260620 03:03:45.429883 18806 heartbeater.cc:464] Registering TS with master...
I20260620 03:03:45.430620 18806 heartbeater.cc:511] Master 127.0.97.62:38623 requested a full tablet report, sending...
I20260620 03:03:45.432772 18617 ts_manager.cc:194] Registered new tserver with Master: e2ffe153162547e9b98c7eee0f1e8a4a (127.0.97.1:41033)
I20260620 03:03:45.434399 18617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51101
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:03:45.468115 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38623
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:03:45.795548 18812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:45.795868 18812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:45.796041 18812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:45.803522 18812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:45.803682 18812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:45.803864 18812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:03:45.813685 18812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:45.815021 18812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:45.816390 18812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:45.830158 18817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:45.830160 18820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:45.830245 18818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:45.831027 18812 server_base.cc:1034] running on GCE node
I20260620 03:03:45.831674 18812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:45.832628 18812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:45.833859 18812 hybrid_clock.cc:648] HybridClock initialized: now 1781924625833809 us; error 65 us; skew 500 ppm
I20260620 03:03:45.834192 18812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:03:45 dist-test-slave-f1j0 krb5kdc[18572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:03:45.841902 18812 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:03:45.843701 18812 webserver.cc:460] Webserver started at http://127.0.97.2:36245/ using document root <none> and password file <none>
I20260620 03:03:45.844417 18812 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:45.844571 18812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:45.844884 18812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:45.847934 18812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "5422739dedbb4382af5fd6108f1b6238"
format_stamp: "Formatted at 2026-06-20 03:03:45 on dist-test-slave-f1j0"
server_key: "84ea648d94cc3a9b97b5545f5418b27e"
server_key_iv: "679bee1144b3277202057bc601b33af2"
server_key_version: "encryptionkey@0"
I20260620 03:03:45.848737 18812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5422739dedbb4382af5fd6108f1b6238"
format_stamp: "Formatted at 2026-06-20 03:03:45 on dist-test-slave-f1j0"
server_key: "84ea648d94cc3a9b97b5545f5418b27e"
server_key_iv: "679bee1144b3277202057bc601b33af2"
server_key_version: "encryptionkey@0"
I20260620 03:03:45.854204 18812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 03:03:45.857771 18827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:45.859066 18812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:03:45.859257 18812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "5422739dedbb4382af5fd6108f1b6238"
format_stamp: "Formatted at 2026-06-20 03:03:45 on dist-test-slave-f1j0"
server_key: "84ea648d94cc3a9b97b5545f5418b27e"
server_key_iv: "679bee1144b3277202057bc601b33af2"
server_key_version: "encryptionkey@0"
I20260620 03:03:45.859472 18812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:45.888931 18812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:45.893013 18812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:45.893325 18812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:45.894348 18812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:45.895705 18812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:45.895810 18812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:45.895958 18812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:45.896039 18812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:45.928867 18812 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44867
I20260620 03:03:45.928951 18939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44867 every 8 connection(s)
I20260620 03:03:45.930253 18812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:03:45.931469 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 18812
I20260620 03:03:45.931643 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:03:45.932226 388 external_mini_cluster.cc:1428] Setting key aec04ea7bee610b1bd9f7e757e329854
Jun 20 03:03:45 dist-test-slave-f1j0 krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:03:45.954656 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:45.933193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49979 (local address 127.0.97.62:38623)
0620 03:03:45.933710 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:45.933715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:45.934695 (+ 980us) server_negotiation.cc:407] Connection header received
0620 03:03:45.937730 (+ 3035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:45.937738 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:45.937825 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:45.937933 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:45.939629 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:45.940854 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:45.942473 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:45.943233 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:45.943357 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:45.947321 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:45.947330 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:45.947338 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:45.947386 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:45.951077 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:45.951852 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:45.951860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:45.951866 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:45.952030 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:45.952487 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:45.952495 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:45.952502 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:45.952900 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:45.953054 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:45.953850 (+ 796us) server_negotiation.cc:299] Negotiation successful
0620 03:03:45.954121 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":262,"threads_started":1}
I20260620 03:03:45.955269 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:45.933669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38623 (local address 127.0.97.2:49979)
0620 03:03:45.934514 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:45.934575 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:45.937294 (+ 2719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:45.938181 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:45.938190 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:45.938719 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:03:45.939446 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:45.939465 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:45.941013 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:45.941017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:45.942326 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:45.942338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:45.943472 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:45.943481 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:45.943831 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:45.944652 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:03:45.944673 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:03:45.947067 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:03:45.951239 (+ 4172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:03:45.951247 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:03:45.951262 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:03:45.951701 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:03:45.952155 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:03:45.952159 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:03:45.952161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:03:45.952348 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:03:45.953054 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:03:45.953060 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:03:45.953402 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 03:03:45.953702 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 03:03:45.954138 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":359,"threads_started":1}
I20260620 03:03:45.956924 18940 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38623
I20260620 03:03:45.957289 18940 heartbeater.cc:464] Registering TS with master...
I20260620 03:03:45.958299 18940 heartbeater.cc:511] Master 127.0.97.62:38623 requested a full tablet report, sending...
I20260620 03:03:45.960094 18617 ts_manager.cc:194] Registered new tserver with Master: 5422739dedbb4382af5fd6108f1b6238 (127.0.97.2:44867)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 03:03:45.961117 18617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:03:45.992202 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38623
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:03:46.320956 18946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:46.321296 18946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:46.321466 18946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:46.328925 18946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:46.329041 18946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:46.329159 18946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:03:46.338912 18946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:46.340276 18946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:46.341602 18946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:46.355304 18951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:46.355228 18952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:46.355265 18954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:46.356316 18946 server_base.cc:1034] running on GCE node
I20260620 03:03:46.357041 18946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:46.358091 18946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:46.359294 18946 hybrid_clock.cc:648] HybridClock initialized: now 1781924626359222 us; error 60 us; skew 500 ppm
I20260620 03:03:46.359656 18946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:03:46 dist-test-slave-f1j0 krb5kdc[18572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:03:46.363129 18946 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:03:46.364723 18946 webserver.cc:460] Webserver started at http://127.0.97.3:35563/ using document root <none> and password file <none>
I20260620 03:03:46.365454 18946 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:46.365622 18946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:46.365940 18946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:46.368849 18946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "f3bb9fa191db4e60a097109b31ef8f7c"
format_stamp: "Formatted at 2026-06-20 03:03:46 on dist-test-slave-f1j0"
server_key: "aedc28c9f0b23828b100bef320139c0d"
server_key_iv: "fc3798cb431f0422710edce41fc1fbae"
server_key_version: "encryptionkey@0"
I20260620 03:03:46.369740 18946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f3bb9fa191db4e60a097109b31ef8f7c"
format_stamp: "Formatted at 2026-06-20 03:03:46 on dist-test-slave-f1j0"
server_key: "aedc28c9f0b23828b100bef320139c0d"
server_key_iv: "fc3798cb431f0422710edce41fc1fbae"
server_key_version: "encryptionkey@0"
I20260620 03:03:46.374869 18946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 03:03:46.378324 18961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:46.379633 18946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:03:46.379819 18946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "f3bb9fa191db4e60a097109b31ef8f7c"
format_stamp: "Formatted at 2026-06-20 03:03:46 on dist-test-slave-f1j0"
server_key: "aedc28c9f0b23828b100bef320139c0d"
server_key_iv: "fc3798cb431f0422710edce41fc1fbae"
server_key_version: "encryptionkey@0"
I20260620 03:03:46.380020 18946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:46.405264 18946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:46.410854 18946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:46.411113 18946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:46.412038 18946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:46.413311 18946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:46.413383 18946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:46.413466 18946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:46.413513 18946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:46.437304 18806 heartbeater.cc:503] Master 127.0.97.62:38623 was elected leader, sending a full tablet report...
I20260620 03:03:46.446794 18946 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33765
I20260620 03:03:46.446880 19073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33765 every 8 connection(s)
I20260620 03:03:46.448132 18946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:03:46.456295 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 18946
I20260620 03:03:46.456485 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:03:46.456940 388 external_mini_cluster.cc:1428] Setting key 84f602e3da9812029b2a94d90a39b627
Jun 20 03:03:46 dist-test-slave-f1j0 krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:03:46.468633 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:46.450800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48055 (local address 127.0.97.62:38623)
0620 03:03:46.450975 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:46.450979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:46.452124 (+ 1145us) server_negotiation.cc:407] Connection header received
0620 03:03:46.454055 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:46.454061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:46.454172 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:46.454285 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:46.455836 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.456678 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.458557 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.459108 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.459244 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:46.462748 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:46.462753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:46.462756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:46.462790 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:46.465543 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:46.466326 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:46.466331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:46.466336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:46.466462 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:46.466901 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:46.466913 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:46.466915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:46.467191 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:46.467349 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:46.468051 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 03:03:46.468281 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 03:03:46.469677 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:46.451256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38623 (local address 127.0.97.3:48055)
0620 03:03:46.451953 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:46.451992 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:46.453777 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:46.454535 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:46.454545 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:46.455060 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:03:46.455632 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:46.455650 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.456824 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.456834 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:46.458409 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:46.458421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.459333 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.459340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:46.459579 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:46.460274 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:03:46.460296 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:03:46.462525 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:03:46.465726 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:03:46.465735 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:03:46.465754 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:03:46.466193 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:03:46.466578 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:03:46.466584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:03:46.466588 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:03:46.466780 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:03:46.467330 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:03:46.467338 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:03:46.467680 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 03:03:46.467944 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 03:03:46.468501 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":272,"threads_started":1}
I20260620 03:03:46.471230 19074 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38623
I20260620 03:03:46.471601 19074 heartbeater.cc:464] Registering TS with master...
I20260620 03:03:46.472466 19074 heartbeater.cc:511] Master 127.0.97.62:38623 requested a full tablet report, sending...
I20260620 03:03:46.473904 18617 ts_manager.cc:194] Registered new tserver with Master: f3bb9fa191db4e60a097109b31ef8f7c (127.0.97.3:33765)
I20260620 03:03:46.474705 18617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48055
I20260620 03:03:46.484694 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:03:46.502287 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:46.488857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54820 (local address 127.0.97.62:38623)
0620 03:03:46.489075 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:46.489084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:46.489467 (+ 383us) server_negotiation.cc:407] Connection header received
0620 03:03:46.489693 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:46.489698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:46.489761 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:46.489864 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:46.491332 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.492075 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.493423 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.493931 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.494072 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:46.496807 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:46.496812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:46.496815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:46.496851 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:46.499342 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:46.500142 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:46.500150 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:46.500161 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:46.500279 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:46.500763 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:46.500770 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:46.500778 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:46.501074 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:46.501383 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:46.501763 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 03:03:46.501949 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 03:03:46.506155 18617 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:03:46.513099 18617 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:03:46.536353 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:46.525205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44867 (local address 127.0.0.1:48616)
0620 03:03:46.527455 (+ 2250us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:46.527494 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:46.527755 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:46.528927 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:46.528937 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:46.528972 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:46.529579 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:46.529597 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.531278 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.531285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:46.533268 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:46.533283 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.535316 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.535321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:46.535511 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:46.535572 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 03:03:46.535696 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 03:03:46.535880 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1974,"mutex_wait_us":62,"thread_start_us":2782,"threads_started":1}
I20260620 03:03:46.536729 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:46.525365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48616 (local address 127.0.97.2:44867)
0620 03:03:46.526030 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:46.526043 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:46.527594 (+ 1551us) server_negotiation.cc:407] Connection header received
0620 03:03:46.528050 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:46.528062 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:46.528367 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:46.528552 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:46.529859 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.531095 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.533438 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.534828 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.534973 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:46.535177 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:46.536248 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0620 03:03:46.536423 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":750,"threads_started":1}
I20260620 03:03:46.536828 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:46.525440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33765 (local address 127.0.0.1:48560)
0620 03:03:46.528260 (+ 2820us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:46.528277 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:46.528416 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:46.529424 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:46.529429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:46.529454 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:46.529955 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:46.529968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.531864 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.531873 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:46.534199 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:46.534213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.535996 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.536004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:46.536182 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:46.536223 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 03:03:46.536309 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 03:03:46.536468 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2684,"mutex_wait_us":25,"thread_start_us":1597,"threads_started":1}
I20260620 03:03:46.537604 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:46.525852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48560 (local address 127.0.97.3:33765)
0620 03:03:46.526529 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:46.526538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:46.528560 (+ 2022us) server_negotiation.cc:407] Connection header received
0620 03:03:46.528661 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:46.528668 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:46.528905 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:46.529094 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:46.530158 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.531700 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.534362 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.535791 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.535927 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:46.536137 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:46.537082 (+ 945us) server_negotiation.cc:299] Negotiation successful
0620 03:03:46.537266 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":432,"threads_started":1}
I20260620 03:03:46.538059 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:46.527144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41033 (local address 127.0.0.1:46244)
0620 03:03:46.529487 (+ 2343us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:46.529507 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:46.529659 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:46.530819 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:46.530824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:46.530848 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:46.531170 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:46.531182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.533307 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.533314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:46.535160 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:46.535176 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.537238 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.537245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:46.537370 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:46.537391 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 03:03:46.537501 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 03:03:46.537647 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2186,"mutex_wait_us":509,"spinlock_wait_cycles":896,"thread_start_us":1484,"threads_started":1}
I20260620 03:03:46.539134 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:46.527514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46244 (local address 127.0.97.1:41033)
0620 03:03:46.530257 (+ 2743us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:46.530268 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:46.530296 (+ 28us) server_negotiation.cc:407] Connection header received
0620 03:03:46.530396 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:46.530404 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:46.530622 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:46.530813 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:46.531593 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.533167 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.535352 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.536753 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.536885 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:46.537110 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:46.537559 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 03:03:46.538829 (+ 1270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2566,"thread_start_us":2409,"threads_started":1}
I20260620 03:03:46.541107 19009 tablet_service.cc:1467] Processing CreateTablet for tablet cc0a6ca9f7c349c2ab5273b63a8288a9 (DEFAULT_TABLE table=test-table [id=9350c928deab43e7884ed82d9a508f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:03:46.541118 18875 tablet_service.cc:1467] Processing CreateTablet for tablet cc0a6ca9f7c349c2ab5273b63a8288a9 (DEFAULT_TABLE table=test-table [id=9350c928deab43e7884ed82d9a508f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:03:46.542389 19009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0a6ca9f7c349c2ab5273b63a8288a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:46.542527 18875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0a6ca9f7c349c2ab5273b63a8288a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:46.542738 18741 tablet_service.cc:1467] Processing CreateTablet for tablet cc0a6ca9f7c349c2ab5273b63a8288a9 (DEFAULT_TABLE table=test-table [id=9350c928deab43e7884ed82d9a508f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:03:46.544042 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0a6ca9f7c349c2ab5273b63a8288a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:46.553840 19092 tablet_bootstrap.cc:492] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c: Bootstrap starting.
I20260620 03:03:46.556622 19093 tablet_bootstrap.cc:492] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238: Bootstrap starting.
I20260620 03:03:46.557667 19092 tablet_bootstrap.cc:654] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:46.559077 19092 log.cc:824] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:46.560264 19093 tablet_bootstrap.cc:654] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:46.561527 19093 log.cc:824] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:46.561690 19094 tablet_bootstrap.cc:492] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a: Bootstrap starting.
I20260620 03:03:46.562376 19092 tablet_bootstrap.cc:492] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c: No bootstrap required, opened a new log
I20260620 03:03:46.562698 19092 ts_tablet_manager.cc:1397] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260620 03:03:46.564694 19093 tablet_bootstrap.cc:492] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238: No bootstrap required, opened a new log
I20260620 03:03:46.564723 19094 tablet_bootstrap.cc:654] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:46.564942 19093 ts_tablet_manager.cc:1397] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 03:03:46.565788 19094 log.cc:824] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:46.571650 19094 tablet_bootstrap.cc:492] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a: No bootstrap required, opened a new log
I20260620 03:03:46.571926 19094 ts_tablet_manager.cc:1397] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20260620 03:03:46.573386 19092 raft_consensus.cc:348] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5422739dedbb4382af5fd6108f1b6238" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44867 } } peers { permanent_uuid: "f3bb9fa191db4e60a097109b31ef8f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "e2ffe153162547e9b98c7eee0f1e8a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41033 } }
I20260620 03:03:46.573856 19092 raft_consensus.cc:374] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:46.573999 19092 raft_consensus.cc:729] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3bb9fa191db4e60a097109b31ef8f7c, State: Initialized, Role: FOLLOWER
I20260620 03:03:46.574642 19092 consensus_queue.cc:260] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5422739dedbb4382af5fd6108f1b6238" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44867 } } peers { permanent_uuid: "f3bb9fa191db4e60a097109b31ef8f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "e2ffe153162547e9b98c7eee0f1e8a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41033 } }
I20260620 03:03:46.575313 19093 raft_consensus.cc:348] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5422739dedbb4382af5fd6108f1b6238" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44867 } } peers { permanent_uuid: "f3bb9fa191db4e60a097109b31ef8f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "e2ffe153162547e9b98c7eee0f1e8a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41033 } }
I20260620 03:03:46.575729 19093 raft_consensus.cc:374] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:46.575736 19074 heartbeater.cc:503] Master 127.0.97.62:38623 was elected leader, sending a full tablet report...
I20260620 03:03:46.575872 19093 raft_consensus.cc:729] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5422739dedbb4382af5fd6108f1b6238, State: Initialized, Role: FOLLOWER
I20260620 03:03:46.576510 19092 ts_tablet_manager.cc:1428] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260620 03:03:46.576476 19093 consensus_queue.cc:260] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5422739dedbb4382af5fd6108f1b6238" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44867 } } peers { permanent_uuid: "f3bb9fa191db4e60a097109b31ef8f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "e2ffe153162547e9b98c7eee0f1e8a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41033 } }
I20260620 03:03:46.578085 18940 heartbeater.cc:503] Master 127.0.97.62:38623 was elected leader, sending a full tablet report...
I20260620 03:03:46.578264 19093 ts_tablet_manager.cc:1428] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
I20260620 03:03:46.584309 19094 raft_consensus.cc:348] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5422739dedbb4382af5fd6108f1b6238" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44867 } } peers { permanent_uuid: "f3bb9fa191db4e60a097109b31ef8f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "e2ffe153162547e9b98c7eee0f1e8a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41033 } }
I20260620 03:03:46.584669 19094 raft_consensus.cc:374] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:46.584764 19094 raft_consensus.cc:729] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ffe153162547e9b98c7eee0f1e8a4a, State: Initialized, Role: FOLLOWER
I20260620 03:03:46.585275 19094 consensus_queue.cc:260] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5422739dedbb4382af5fd6108f1b6238" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44867 } } peers { permanent_uuid: "f3bb9fa191db4e60a097109b31ef8f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "e2ffe153162547e9b98c7eee0f1e8a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41033 } }
I20260620 03:03:46.586829 19094 ts_tablet_manager.cc:1428] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a: Time spent starting tablet: real 0.015s user 0.004s sys 0.008s
W20260620 03:03:46.657199 18807 tablet.cc:2369] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:03:46.682272 18941 tablet.cc:2369] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:03:46.699872 19075 tablet.cc:2369] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:03:46.711992 19099 raft_consensus.cc:482] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:03:46.712352 19099 raft_consensus.cc:504] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5422739dedbb4382af5fd6108f1b6238" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44867 } } peers { permanent_uuid: "f3bb9fa191db4e60a097109b31ef8f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "e2ffe153162547e9b98c7eee0f1e8a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41033 } }
I20260620 03:03:46.714367 19099 leader_election.cc:288] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3bb9fa191db4e60a097109b31ef8f7c (127.0.97.3:33765), e2ffe153162547e9b98c7eee0f1e8a4a (127.0.97.1:41033)
I20260620 03:03:46.721967 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:46.714913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54705 (local address 127.0.97.3:33765)
0620 03:03:46.715111 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:46.715118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:46.715685 (+ 567us) server_negotiation.cc:407] Connection header received
0620 03:03:46.715958 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:46.715964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:46.716051 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:46.716159 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:46.716803 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.717922 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.720023 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.720874 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.720973 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:46.721064 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:46.721532 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 03:03:46.721707 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 03:03:46.722142 19101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:46.714850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33765 (local address 127.0.97.2:54705)
0620 03:03:46.715566 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:46.715582 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:46.715762 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:46.716200 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:46.716207 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:46.716242 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:46.716623 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:46.716654 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.718060 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.718066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:46.719868 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:46.719880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.721118 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.721125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:46.721312 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:46.721354 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 03:03:46.721473 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 03:03:46.721651 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":2,"thread_start_us":413,"threads_started":1}
I20260620 03:03:46.722592 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:46.714850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35977 (local address 127.0.97.1:41033)
0620 03:03:46.715061 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:46.715068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:46.716319 (+ 1251us) server_negotiation.cc:407] Connection header received
0620 03:03:46.716520 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:46.716524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:46.716583 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:46.716653 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:46.717178 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.718191 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.720334 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.721378 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.721614 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:46.721730 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:46.722128 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 03:03:46.722286 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 03:03:46.722844 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:46.714850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41033 (local address 127.0.97.2:35977)
0620 03:03:46.716195 (+ 1345us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:46.716210 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:46.716389 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:46.716689 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:46.716696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:46.716727 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:46.717020 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:46.717028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.718321 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.718326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:46.720118 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:46.720133 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.721735 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.721742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:46.721925 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:46.721965 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 03:03:46.722069 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 03:03:46.722207 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1207,"mutex_wait_us":48,"spinlock_wait_cycles":25984,"thread_start_us":955,"threads_started":1}
I20260620 03:03:46.723311 19029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0a6ca9f7c349c2ab5273b63a8288a9" candidate_uuid: "5422739dedbb4382af5fd6108f1b6238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3bb9fa191db4e60a097109b31ef8f7c" is_pre_election: true
I20260620 03:03:46.723843 19029 raft_consensus.cc:2454] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5422739dedbb4382af5fd6108f1b6238 in term 0.
I20260620 03:03:46.723887 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0a6ca9f7c349c2ab5273b63a8288a9" candidate_uuid: "5422739dedbb4382af5fd6108f1b6238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ffe153162547e9b98c7eee0f1e8a4a" is_pre_election: true
I20260620 03:03:46.724416 18761 raft_consensus.cc:2454] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5422739dedbb4382af5fd6108f1b6238 in term 0.
I20260620 03:03:46.724545 18830 leader_election.cc:302] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5422739dedbb4382af5fd6108f1b6238, f3bb9fa191db4e60a097109b31ef8f7c; no voters:
I20260620 03:03:46.724992 19099 raft_consensus.cc:2781] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:03:46.725119 19099 raft_consensus.cc:482] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:03:46.725271 19099 raft_consensus.cc:3037] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:46.727679 19099 raft_consensus.cc:504] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5422739dedbb4382af5fd6108f1b6238" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44867 } } peers { permanent_uuid: "f3bb9fa191db4e60a097109b31ef8f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "e2ffe153162547e9b98c7eee0f1e8a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41033 } }
I20260620 03:03:46.728325 19099 leader_election.cc:288] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [CANDIDATE]: Term 1 election: Requested vote from peers f3bb9fa191db4e60a097109b31ef8f7c (127.0.97.3:33765), e2ffe153162547e9b98c7eee0f1e8a4a (127.0.97.1:41033)
I20260620 03:03:46.728804 19029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0a6ca9f7c349c2ab5273b63a8288a9" candidate_uuid: "5422739dedbb4382af5fd6108f1b6238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3bb9fa191db4e60a097109b31ef8f7c"
I20260620 03:03:46.728850 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0a6ca9f7c349c2ab5273b63a8288a9" candidate_uuid: "5422739dedbb4382af5fd6108f1b6238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ffe153162547e9b98c7eee0f1e8a4a"
I20260620 03:03:46.729010 19029 raft_consensus.cc:3037] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:46.729068 18761 raft_consensus.cc:3037] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:46.731350 18761 raft_consensus.cc:2454] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5422739dedbb4382af5fd6108f1b6238 in term 1.
I20260620 03:03:46.731597 19029 raft_consensus.cc:2454] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5422739dedbb4382af5fd6108f1b6238 in term 1.
I20260620 03:03:46.731880 18829 leader_election.cc:302] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5422739dedbb4382af5fd6108f1b6238, e2ffe153162547e9b98c7eee0f1e8a4a; no voters:
I20260620 03:03:46.732219 19099 raft_consensus.cc:2781] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:03:46.732658 19099 raft_consensus.cc:686] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [term 1 LEADER]: Becoming Leader. State: Replica: 5422739dedbb4382af5fd6108f1b6238, State: Running, Role: LEADER
I20260620 03:03:46.733131 19099 consensus_queue.cc:237] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5422739dedbb4382af5fd6108f1b6238" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44867 } } peers { permanent_uuid: "f3bb9fa191db4e60a097109b31ef8f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "e2ffe153162547e9b98c7eee0f1e8a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41033 } }
I20260620 03:03:46.738267 18616 catalog_manager.cc:5495] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5422739dedbb4382af5fd6108f1b6238 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "5422739dedbb4382af5fd6108f1b6238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5422739dedbb4382af5fd6108f1b6238" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3bb9fa191db4e60a097109b31ef8f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2ffe153162547e9b98c7eee0f1e8a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41033 } health_report { overall_health: UNKNOWN } } }
I20260620 03:03:46.777061 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:46.771844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48622 (local address 127.0.97.2:44867)
0620 03:03:46.772027 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:46.772031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:46.772103 (+ 72us) server_negotiation.cc:407] Connection header received
0620 03:03:46.772284 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:46.772287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:46.772376 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:46.772502 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:03:46.772841 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.773552 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.775010 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.775417 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.775505 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:46.776025 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:03:46.776535 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:03:46.776664 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:46.776765 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 03:03:46.776806 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 03:03:46.786912 18761 raft_consensus.cc:1261] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a [term 1 FOLLOWER]: Refusing update from remote peer 5422739dedbb4382af5fd6108f1b6238: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:03:46.786926 19029 raft_consensus.cc:1261] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c [term 1 FOLLOWER]: Refusing update from remote peer 5422739dedbb4382af5fd6108f1b6238: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:03:46.787961 19103 consensus_queue.cc:1035] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3bb9fa191db4e60a097109b31ef8f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:03:46.788329 19099 consensus_queue.cc:1035] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2ffe153162547e9b98c7eee0f1e8a4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:03:46.800494 19106 mvcc.cc:205] Tried to move back new op lower bound from 7298763271307632640 to 7298763271102902272. Current Snapshot: MvccSnapshot[applied={T|T < 7298763271307632640}]
I20260620 03:03:46.819093 18616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:03:46.819406 18616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:03:46.823772 18616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2543967755574e85a2f88924b22e7231: Sending DeleteTablet for 3 replicas of tablet cc0a6ca9f7c349c2ab5273b63a8288a9
I20260620 03:03:46.824954 18875 tablet_service.cc:1514] Processing DeleteTablet for tablet cc0a6ca9f7c349c2ab5273b63a8288a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:03:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48616
I20260620 03:03:46.824947 18741 tablet_service.cc:1514] Processing DeleteTablet for tablet cc0a6ca9f7c349c2ab5273b63a8288a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:03:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46244
I20260620 03:03:46.825356 19009 tablet_service.cc:1514] Processing DeleteTablet for tablet cc0a6ca9f7c349c2ab5273b63a8288a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:03:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48560
I20260620 03:03:46.828420 18740 tablet_service.cc:1514] Processing DeleteTablet for tablet cc0a6ca9f7c349c2ab5273b63a8288a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:03:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46244
I20260620 03:03:46.831094 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 18678
I20260620 03:03:46.837044 19116 tablet_replica.cc:330] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238: stopping tablet replica
I20260620 03:03:46.837668 19116 raft_consensus.cc:2229] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:03:46.838531 19116 raft_consensus.cc:2258] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:03:46.839829 19117 tablet_replica.cc:330] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a: stopping tablet replica
I20260620 03:03:46.840222 19117 raft_consensus.cc:2229] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:03:46.840665 19117 raft_consensus.cc:2258] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:03:46.842177 19116 ts_tablet_manager.cc:1905] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:03:46.844756 19118 tablet_replica.cc:330] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c: stopping tablet replica
I20260620 03:03:46.845156 19118 raft_consensus.cc:2229] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:03:46.847438 19117 ts_tablet_manager.cc:1905] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:03:46.848881 19116 ts_tablet_manager.cc:1918] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:03:46.849047 19116 log.cc:1196] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/cc0a6ca9f7c349c2ab5273b63a8288a9
I20260620 03:03:46.849668 19116 ts_tablet_manager.cc:1939] T cc0a6ca9f7c349c2ab5273b63a8288a9 P 5422739dedbb4382af5fd6108f1b6238: Deleting consensus metadata
I20260620 03:03:46.850605 19118 raft_consensus.cc:2258] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:03:46.851662 18604 catalog_manager.cc:4841] TS 5422739dedbb4382af5fd6108f1b6238 (127.0.97.2:44867): tablet cc0a6ca9f7c349c2ab5273b63a8288a9 (table test-table [id=9350c928deab43e7884ed82d9a508f09]) successfully deleted
I20260620 03:03:46.853485 19118 ts_tablet_manager.cc:1905] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:03:46.854848 19117 ts_tablet_manager.cc:1918] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:03:46.855082 19117 log.cc:1196] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/cc0a6ca9f7c349c2ab5273b63a8288a9
I20260620 03:03:46.855633 19117 ts_tablet_manager.cc:1939] T cc0a6ca9f7c349c2ab5273b63a8288a9 P e2ffe153162547e9b98c7eee0f1e8a4a: Deleting consensus metadata
Jun 20 03:03:46 dist-test-slave-f1j0 krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:03:46.857537 18602 catalog_manager.cc:4841] TS e2ffe153162547e9b98c7eee0f1e8a4a (127.0.97.1:41033): tablet cc0a6ca9f7c349c2ab5273b63a8288a9 (table test-table [id=9350c928deab43e7884ed82d9a508f09]) successfully deleted
W20260620 03:03:46.858716 18602 catalog_manager.cc:4808] TS e2ffe153162547e9b98c7eee0f1e8a4a (127.0.97.1:41033): delete failed for tablet cc0a6ca9f7c349c2ab5273b63a8288a9 because the tablet was not found. No further retry: Not found: Tablet not found: cc0a6ca9f7c349c2ab5273b63a8288a9
I20260620 03:03:46.860097 19118 ts_tablet_manager.cc:1918] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:03:46.860291 19118 log.cc:1196] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/cc0a6ca9f7c349c2ab5273b63a8288a9
I20260620 03:03:46.860756 19118 ts_tablet_manager.cc:1939] T cc0a6ca9f7c349c2ab5273b63a8288a9 P f3bb9fa191db4e60a097109b31ef8f7c: Deleting consensus metadata
I20260620 03:03:46.862461 18603 catalog_manager.cc:4841] TS f3bb9fa191db4e60a097109b31ef8f7c (127.0.97.3:33765): tablet cc0a6ca9f7c349c2ab5273b63a8288a9 (table test-table [id=9350c928deab43e7884ed82d9a508f09]) successfully deleted
I20260620 03:03:46.864864 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:46.832356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46260 (local address 127.0.97.1:41033)
0620 03:03:46.832563 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:46.832573 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:46.843894 (+ 11321us) server_negotiation.cc:407] Connection header received
0620 03:03:46.844115 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:46.844121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:46.844214 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:46.844347 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:46.847115 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.848248 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.850995 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:46.851688 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:46.851786 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:46.857541 (+ 5755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:46.857558 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:46.857617 (+ 59us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:46.857665 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:46.861625 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:46.862548 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:46.862557 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:46.862560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:46.862748 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:46.863194 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:46.863200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:46.863203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:46.863561 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:46.863888 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:46.864283 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 03:03:46.864530 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 03:03:46.865530 18801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:47.002934 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 18678
I20260620 03:03:47.020665 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 18812
Jun 20 03:03:47 dist-test-slave-f1j0 krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:03:47.038561 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:47.021693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48626 (local address 127.0.97.2:44867)
0620 03:03:47.021866 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:47.021871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:47.021891 (+ 20us) server_negotiation.cc:407] Connection header received
0620 03:03:47.021948 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:47.021951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:47.022012 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:47.022109 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:47.023912 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:47.024720 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:47.026595 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:47.027009 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:47.027110 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:47.032141 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:47.032155 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:47.032171 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:47.032203 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:47.035462 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:47.036194 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:47.036201 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:47.036203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:47.036338 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:47.036749 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:47.036753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:47.036755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:47.037174 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:47.037476 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:47.037917 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 03:03:47.038156 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 03:03:47.039256 18935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:47.155471 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 18812
I20260620 03:03:47.174664 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 18946
Jun 20 03:03:47 dist-test-slave-f1j0 krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:03:47.189296 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:47.175381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48566 (local address 127.0.97.3:33765)
0620 03:03:47.175623 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:47.175628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:47.175646 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:03:47.175734 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:47.175738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:47.175797 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:47.175876 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:47.177049 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:47.177830 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:47.179117 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:47.179480 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:47.179574 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:47.183846 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:47.183860 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:47.183876 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:47.183914 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:47.186819 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:47.187470 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:47.187477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:47.187485 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:47.187661 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:47.187966 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:47.187969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:47.187971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:47.188321 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:47.188531 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:47.188748 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 03:03:47.188917 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 03:03:47.190068 19069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:47.278864 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 18946
I20260620 03:03:47.295960 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 18585
I20260620 03:03:47.297006 18647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:47.406904 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 18585
2026-06-20T03:03:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4409 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T03:03:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:03:47Z Disabled control of system clock
I20260620 03:03:47.482467 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40143
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 03:03:47.819216 19127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:47.819533 19127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:47.819617 19127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:47.826906 19127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:03:47.827006 19127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:47.827044 19127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:47.827093 19127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:03:47.827131 19127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:03:47.836122 19127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40143
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:47.837492 19127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:47.839222 19127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:47.851341 19133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:47.851464 19132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:47.851442 19135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:47.852165 19127 server_base.cc:1034] running on GCE node
I20260620 03:03:47.852972 19127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:47.854470 19127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:47.855684 19127 hybrid_clock.cc:648] HybridClock initialized: now 1781924627855640 us; error 30 us; skew 500 ppm
I20260620 03:03:47.856053 19127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:03:47.858480 19127 webserver.cc:460] Webserver started at http://127.0.97.62:35687/ using document root <none> and password file <none>
I20260620 03:03:47.859176 19127 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:47.859292 19127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:47.859630 19127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:47.862596 19127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "6e17cb505ffb4b2b83ddbd7916e6f111"
format_stamp: "Formatted at 2026-06-20 03:03:47 on dist-test-slave-f1j0"
server_key: "90601e2fdd7108d4fa0adf9517bef69d"
server_key_iv: "635659c2c36a4d0874a88239f740e650"
server_key_version: "encryptionkey@0"
I20260620 03:03:47.863401 19127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "6e17cb505ffb4b2b83ddbd7916e6f111"
format_stamp: "Formatted at 2026-06-20 03:03:47 on dist-test-slave-f1j0"
server_key: "90601e2fdd7108d4fa0adf9517bef69d"
server_key_iv: "635659c2c36a4d0874a88239f740e650"
server_key_version: "encryptionkey@0"
I20260620 03:03:47.868494 19127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:03:47.871905 19141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:47.873412 19127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 03:03:47.873631 19127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "6e17cb505ffb4b2b83ddbd7916e6f111"
format_stamp: "Formatted at 2026-06-20 03:03:47 on dist-test-slave-f1j0"
server_key: "90601e2fdd7108d4fa0adf9517bef69d"
server_key_iv: "635659c2c36a4d0874a88239f740e650"
server_key_version: "encryptionkey@0"
I20260620 03:03:47.873864 19127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:47.894773 19127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:47.895666 19127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:47.895972 19127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:47.914336 19127 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40143
I20260620 03:03:47.914355 19192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40143 every 8 connection(s)
I20260620 03:03:47.915661 19127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:03:47.918156 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 19127
I20260620 03:03:47.918331 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:03:47.918781 388 external_mini_cluster.cc:1428] Setting key ba4a3405f75b22fed020f5bf3d94dcb7
I20260620 03:03:47.919613 19193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:47.931932 19196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:47.920486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.0.97.62:40143)
0620 03:03:47.921572 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:47.921610 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:47.921678 (+ 68us) server_negotiation.cc:407] Connection header received
0620 03:03:47.923331 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:47.923349 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:47.923807 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:47.924204 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:47.924752 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:47.926133 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:47.927608 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:47.928273 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:47.928475 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:47.929716 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:47.929727 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:47.929740 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:47.930302 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:47.930377 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:47.930400 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:47.930540 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:47.930746 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 03:03:47.931037 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":353,"threads_started":1}
I20260620 03:03:47.932194 19193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111: Bootstrap starting.
I20260620 03:03:47.935461 19193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:47.936444 19193 log.cc:824] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:47.939054 19193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111: No bootstrap required, opened a new log
I20260620 03:03:47.947160 19193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e17cb505ffb4b2b83ddbd7916e6f111" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40143 } }
I20260620 03:03:47.947634 19193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:47.947743 19193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e17cb505ffb4b2b83ddbd7916e6f111, State: Initialized, Role: FOLLOWER
I20260620 03:03:47.948283 19193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e17cb505ffb4b2b83ddbd7916e6f111" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40143 } }
I20260620 03:03:47.948523 19193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:03:47.948684 19193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:03:47.948863 19193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:47.951092 19193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e17cb505ffb4b2b83ddbd7916e6f111" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40143 } }
I20260620 03:03:47.951592 19193 leader_election.cc:302] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e17cb505ffb4b2b83ddbd7916e6f111; no voters:
I20260620 03:03:47.952178 19193 leader_election.cc:288] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:03:47.952350 19198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:03:47.953111 19198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111 [term 1 LEADER]: Becoming Leader. State: Replica: 6e17cb505ffb4b2b83ddbd7916e6f111, State: Running, Role: LEADER
I20260620 03:03:47.953723 19198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e17cb505ffb4b2b83ddbd7916e6f111" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40143 } }
I20260620 03:03:47.954337 19193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:03:47.956697 19200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e17cb505ffb4b2b83ddbd7916e6f111. Latest consensus state: current_term: 1 leader_uuid: "6e17cb505ffb4b2b83ddbd7916e6f111" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e17cb505ffb4b2b83ddbd7916e6f111" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40143 } } }
I20260620 03:03:47.957120 19200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111 [sys.catalog]: This master's current role is: LEADER
I20260620 03:03:47.956724 19199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e17cb505ffb4b2b83ddbd7916e6f111" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e17cb505ffb4b2b83ddbd7916e6f111" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40143 } } }
I20260620 03:03:47.958756 19199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111 [sys.catalog]: This master's current role is: LEADER
I20260620 03:03:47.962594 19204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:03:47.969648 19204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:03:47.979180 19204 catalog_manager.cc:1347] Generated new cluster ID: a9e4084de1274eb1993f727ba2ce87fa
I20260620 03:03:47.979307 19204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:03:48.006445 19204 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:03:48.007784 19204 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:03:48.017035 19204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111: Generated new TSK 0
I20260620 03:03:48.017938 19204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:03:48.029726 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40143
--builtin_ntp_servers=127.0.97.20:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 03:03:48.365393 19217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:48.365756 19217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:48.365918 19217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:48.374508 19217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:48.374624 19217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:48.374853 19217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:03:48.385102 19217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:48.386514 19217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:48.388305 19217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:48.401958 19223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:48.402110 19222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:48.402218 19225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:48.403522 19217 server_base.cc:1034] running on GCE node
I20260620 03:03:48.404189 19217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:48.405217 19217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:48.406458 19217 hybrid_clock.cc:648] HybridClock initialized: now 1781924628406402 us; error 56 us; skew 500 ppm
I20260620 03:03:48.406805 19217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:03:48.409138 19217 webserver.cc:460] Webserver started at http://127.0.97.1:36673/ using document root <none> and password file <none>
I20260620 03:03:48.409977 19217 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:48.410120 19217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:48.410467 19217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:48.413848 19217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "7e6d593f87074c02b55aa2d394bcf5a0"
format_stamp: "Formatted at 2026-06-20 03:03:48 on dist-test-slave-f1j0"
server_key: "716b4b7dd812b66792cf453cb1efb881"
server_key_iv: "37bfc96222d83e303d061dd5cd9b2e8c"
server_key_version: "encryptionkey@0"
I20260620 03:03:48.414736 19217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7e6d593f87074c02b55aa2d394bcf5a0"
format_stamp: "Formatted at 2026-06-20 03:03:48 on dist-test-slave-f1j0"
server_key: "716b4b7dd812b66792cf453cb1efb881"
server_key_iv: "37bfc96222d83e303d061dd5cd9b2e8c"
server_key_version: "encryptionkey@0"
I20260620 03:03:48.420042 19217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:03:48.423694 19231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:48.425071 19217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 03:03:48.425254 19217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "7e6d593f87074c02b55aa2d394bcf5a0"
format_stamp: "Formatted at 2026-06-20 03:03:48 on dist-test-slave-f1j0"
server_key: "716b4b7dd812b66792cf453cb1efb881"
server_key_iv: "37bfc96222d83e303d061dd5cd9b2e8c"
server_key_version: "encryptionkey@0"
I20260620 03:03:48.425495 19217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:48.485128 19217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:48.486104 19217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:48.486402 19217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:48.487409 19217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:48.488921 19217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:48.489046 19217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:48.489185 19217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:48.489275 19217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:48.523186 19217 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33685
I20260620 03:03:48.523239 19343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33685 every 8 connection(s)
I20260620 03:03:48.524561 19217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:03:48.527022 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 19217
I20260620 03:03:48.527190 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:03:48.527716 388 external_mini_cluster.cc:1428] Setting key 5b416157f2389c4db8e56f169bc592ab
I20260620 03:03:48.531038 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40143
--builtin_ntp_servers=127.0.97.20:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 03:03:48.539113 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:48.527199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36273 (local address 127.0.97.62:40143)
0620 03:03:48.527689 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:48.527694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:48.529325 (+ 1631us) server_negotiation.cc:407] Connection header received
0620 03:03:48.531505 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:48.531514 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:48.531621 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:48.531767 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:48.532836 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:48.534142 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:48.535681 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:48.536520 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:48.536669 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:48.538077 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:48.538087 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:48.538096 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:48.538124 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:48.538182 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:48.538201 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:48.538303 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:48.538505 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 03:03:48.538622 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":235,"threads_started":1}
I20260620 03:03:48.539930 19347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:48.527931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40143 (local address 127.0.97.1:36273)
0620 03:03:48.529106 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:48.529208 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:48.531200 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:48.531981 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:48.531990 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:48.532092 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:03:48.532656 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:48.532673 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:48.534302 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:48.534306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:48.535526 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:48.535538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:48.536723 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:48.536728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:48.536922 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:48.537743 (+ 821us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:03:48.537745 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:03:48.537787 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:03:48.537864 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:03:48.538303 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:03:48.538312 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:03:48.538317 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 03:03:48.538455 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 03:03:48.538737 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":353,"threads_started":1}
I20260620 03:03:48.541527 19344 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40143
I20260620 03:03:48.541858 19344 heartbeater.cc:464] Registering TS with master...
I20260620 03:03:48.542732 19344 heartbeater.cc:511] Master 127.0.97.62:40143 requested a full tablet report, sending...
I20260620 03:03:48.544857 19158 ts_manager.cc:194] Registered new tserver with Master: 7e6d593f87074c02b55aa2d394bcf5a0 (127.0.97.1:33685)
I20260620 03:03:48.546375 19158 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:36273
W20260620 03:03:48.862138 19348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:48.862416 19348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:48.862567 19348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:48.870227 19348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:48.870325 19348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:48.870522 19348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:03:48.879873 19348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:48.881191 19348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:48.883000 19348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:48.896282 19353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:48.896245 19356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:48.896315 19354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:48.896870 19348 server_base.cc:1034] running on GCE node
I20260620 03:03:48.897516 19348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:48.898509 19348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:48.899742 19348 hybrid_clock.cc:648] HybridClock initialized: now 1781924628899698 us; error 58 us; skew 500 ppm
I20260620 03:03:48.900151 19348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:03:48.902427 19348 webserver.cc:460] Webserver started at http://127.0.97.2:41153/ using document root <none> and password file <none>
I20260620 03:03:48.903079 19348 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:48.903163 19348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:48.903434 19348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:48.906224 19348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "cd6d9b3c01a94ef6816deb046082f565"
format_stamp: "Formatted at 2026-06-20 03:03:48 on dist-test-slave-f1j0"
server_key: "c2becbc93ea64a83897a9e61d0a162e3"
server_key_iv: "0182a290a57f29c83e6fd21fe595b714"
server_key_version: "encryptionkey@0"
I20260620 03:03:48.907038 19348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "cd6d9b3c01a94ef6816deb046082f565"
format_stamp: "Formatted at 2026-06-20 03:03:48 on dist-test-slave-f1j0"
server_key: "c2becbc93ea64a83897a9e61d0a162e3"
server_key_iv: "0182a290a57f29c83e6fd21fe595b714"
server_key_version: "encryptionkey@0"
I20260620 03:03:48.911968 19348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:03:48.915472 19362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:48.916851 19348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 03:03:48.917028 19348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "cd6d9b3c01a94ef6816deb046082f565"
format_stamp: "Formatted at 2026-06-20 03:03:48 on dist-test-slave-f1j0"
server_key: "c2becbc93ea64a83897a9e61d0a162e3"
server_key_iv: "0182a290a57f29c83e6fd21fe595b714"
server_key_version: "encryptionkey@0"
I20260620 03:03:48.917243 19348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:48.946997 19348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:48.947857 19348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:48.948150 19348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:48.949120 19348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:48.950637 19348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:48.950759 19348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:48.950897 19348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:48.950985 19348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:48.984261 19348 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33285
I20260620 03:03:48.984354 19474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33285 every 8 connection(s)
I20260620 03:03:48.985643 19348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:03:48.989480 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 19348
I20260620 03:03:48.989670 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:03:48.990243 388 external_mini_cluster.cc:1428] Setting key e894e1e3148c60a9a350b44bfa8b48c9
I20260620 03:03:48.993083 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40143
--builtin_ntp_servers=127.0.97.20:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 03:03:48.999096 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:48.988368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36969 (local address 127.0.97.62:40143)
0620 03:03:48.988541 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:48.988561 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:48.990191 (+ 1630us) server_negotiation.cc:407] Connection header received
0620 03:03:48.992516 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:48.992522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:48.992616 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:48.992724 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:48.993934 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:48.994650 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:48.996209 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:48.996758 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:48.996888 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:48.998288 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:48.998294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:48.998298 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:48.998325 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:48.998357 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:48.998369 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:48.998466 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:48.998696 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 03:03:48.998791 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:03:49.000123 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:48.988905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40143 (local address 127.0.97.2:36969)
0620 03:03:48.990026 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:48.990080 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:48.992207 (+ 2127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:48.992961 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:48.992969 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:48.993066 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:03:48.993769 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:48.993792 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:48.994802 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:48.994808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:48.996057 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:48.996069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:48.996978 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:48.996985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:48.997214 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:48.997966 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:03:48.997968 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:03:48.998012 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:03:48.998095 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:03:48.998464 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:03:48.998475 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:03:48.998485 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 03:03:48.998671 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 03:03:48.999064 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":363,"threads_started":1}
I20260620 03:03:49.001551 19475 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40143
I20260620 03:03:49.001871 19475 heartbeater.cc:464] Registering TS with master...
I20260620 03:03:49.002592 19475 heartbeater.cc:511] Master 127.0.97.62:40143 requested a full tablet report, sending...
I20260620 03:03:49.003990 19158 ts_manager.cc:194] Registered new tserver with Master: cd6d9b3c01a94ef6816deb046082f565 (127.0.97.2:33285)
I20260620 03:03:49.004798 19158 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:36969
W20260620 03:03:49.322918 19478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:49.323230 19478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:49.323393 19478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:49.330693 19478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:49.330828 19478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:49.331007 19478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:03:49.339993 19478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:49.341290 19478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:49.343056 19478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:49.356280 19483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:49.356217 19484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:49.356417 19486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:49.356895 19478 server_base.cc:1034] running on GCE node
I20260620 03:03:49.357643 19478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:49.358595 19478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:49.359825 19478 hybrid_clock.cc:648] HybridClock initialized: now 1781924629359777 us; error 59 us; skew 500 ppm
I20260620 03:03:49.360157 19478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:03:49.362412 19478 webserver.cc:460] Webserver started at http://127.0.97.3:38135/ using document root <none> and password file <none>
I20260620 03:03:49.363104 19478 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:49.363233 19478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:49.363564 19478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:49.366644 19478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "9d8cd812c6204190852cabe030585b22"
format_stamp: "Formatted at 2026-06-20 03:03:49 on dist-test-slave-f1j0"
server_key: "8a7c16c9c44b77110eb1cf7e4355b613"
server_key_iv: "3fb7424803fb6733094a4d8428fdd07d"
server_key_version: "encryptionkey@0"
I20260620 03:03:49.367396 19478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9d8cd812c6204190852cabe030585b22"
format_stamp: "Formatted at 2026-06-20 03:03:49 on dist-test-slave-f1j0"
server_key: "8a7c16c9c44b77110eb1cf7e4355b613"
server_key_iv: "3fb7424803fb6733094a4d8428fdd07d"
server_key_version: "encryptionkey@0"
I20260620 03:03:49.374550 19478 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 03:03:49.378719 19492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:49.379977 19478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 03:03:49.380146 19478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "9d8cd812c6204190852cabe030585b22"
format_stamp: "Formatted at 2026-06-20 03:03:49 on dist-test-slave-f1j0"
server_key: "8a7c16c9c44b77110eb1cf7e4355b613"
server_key_iv: "3fb7424803fb6733094a4d8428fdd07d"
server_key_version: "encryptionkey@0"
I20260620 03:03:49.380367 19478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:49.394182 19478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:49.394977 19478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:49.395249 19478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:49.396216 19478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:49.397851 19478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:49.397977 19478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:49.398106 19478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:49.398197 19478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:49.431247 19478 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40019
I20260620 03:03:49.431331 19604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40019 every 8 connection(s)
I20260620 03:03:49.432610 19478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:03:49.437930 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 19478
I20260620 03:03:49.438117 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:03:49.438540 388 external_mini_cluster.cc:1428] Setting key a0563ce3ee615d3b249be554697f9c39
I20260620 03:03:49.445690 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:49.435312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59903 (local address 127.0.97.62:40143)
0620 03:03:49.435495 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:49.435500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:49.436859 (+ 1359us) server_negotiation.cc:407] Connection header received
0620 03:03:49.438802 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:49.438808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:49.438897 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:49.439018 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:49.440521 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.441434 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.442994 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.443392 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.443526 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:49.444830 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:49.444835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:49.444838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:49.444854 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:49.444884 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:49.444895 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:49.444996 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:49.445209 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 03:03:49.445303 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 03:03:49.446664 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:49.435852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40143 (local address 127.0.97.3:59903)
0620 03:03:49.436726 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:49.436761 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:49.438494 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:49.439512 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:49.439521 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:49.439646 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:03:49.440348 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:49.440374 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.441600 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.441605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:49.442841 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:49.442852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.443616 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.443623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:49.443844 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:49.444511 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:03:49.444513 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:03:49.444557 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:03:49.444630 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:03:49.444993 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:03:49.445004 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:03:49.445013 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 03:03:49.445200 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 03:03:49.445609 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":272,"threads_started":1}
I20260620 03:03:49.447979 19605 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40143
I20260620 03:03:49.448307 19605 heartbeater.cc:464] Registering TS with master...
I20260620 03:03:49.448994 19605 heartbeater.cc:511] Master 127.0.97.62:40143 requested a full tablet report, sending...
I20260620 03:03:49.450397 19158 ts_manager.cc:194] Registered new tserver with Master: 9d8cd812c6204190852cabe030585b22 (127.0.97.3:40019)
I20260620 03:03:49.451179 19158 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:59903
I20260620 03:03:49.453420 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:03:49.463447 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:49.457702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48014 (local address 127.0.97.62:40143)
0620 03:03:49.457946 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:49.457956 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:49.458293 (+ 337us) server_negotiation.cc:407] Connection header received
0620 03:03:49.458419 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:49.458423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:49.458488 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:49.458570 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:49.459060 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.459697 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.461030 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.461425 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.461532 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:49.462752 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:49.462757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:49.462760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:49.462777 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:49.462806 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:49.462816 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:49.462917 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:49.463078 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 03:03:49.463153 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 03:03:49.467384 19158 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:03:49.474299 19158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:03:49.499444 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:49.486311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33285 (local address 127.0.0.1:58256)
0620 03:03:49.487366 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:49.487408 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:49.487574 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:49.490501 (+ 2927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:49.490509 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:49.490547 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:49.491655 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:49.491673 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.493940 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.493946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:49.496483 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:49.496509 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.498481 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.498487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:49.498664 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:49.498728 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 03:03:49.498799 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 03:03:49.499024 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":1,"thread_start_us":653,"threads_started":1}
I20260620 03:03:49.499490 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:49.490120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44330 (local address 127.0.97.1:33685)
0620 03:03:49.490650 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:49.490657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:49.492301 (+ 1644us) server_negotiation.cc:407] Connection header received
0620 03:03:49.492444 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:49.492449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:49.492629 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:49.492808 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:49.493301 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.494688 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.497178 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.498205 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.498368 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:49.498634 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:49.498957 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 03:03:49.499160 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":249,"threads_started":1}
I20260620 03:03:49.500136 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:49.490676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33685 (local address 127.0.0.1:44330)
0620 03:03:49.492184 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:49.492200 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:49.492326 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:49.492811 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:49.492815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:49.492839 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:49.493156 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:49.493166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.494861 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.494871 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:49.496991 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:49.497009 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.498400 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.498406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:49.498609 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:49.498660 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 03:03:49.498819 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 03:03:49.499784 (+ 965us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1403,"mutex_wait_us":817,"thread_start_us":1280,"threads_started":1}
I20260620 03:03:49.500186 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:49.488027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.0.97.3:40019)
0620 03:03:49.489436 (+ 1409us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:49.489448 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:49.489880 (+ 432us) server_negotiation.cc:407] Connection header received
0620 03:03:49.490064 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:49.490073 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:49.490344 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:49.490558 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:49.492848 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.494259 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.496786 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.498259 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.498398 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:49.498667 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:49.499669 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0620 03:03:49.499836 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1227,"thread_start_us":1084,"threads_started":1}
I20260620 03:03:49.499538 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:49.486565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40019 (local address 127.0.0.1:60564)
0620 03:03:49.489752 (+ 3187us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:49.489771 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:49.489894 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:49.490732 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:49.490736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:49.490757 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:49.492640 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:49.492653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.494390 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.494397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:49.496510 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:49.496522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.498790 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.498797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:49.498980 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:49.499009 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 03:03:49.499095 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 03:03:49.499221 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3044,"mutex_wait_us":223,"thread_start_us":2893,"threads_started":1}
I20260620 03:03:49.503345 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:49.486533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58256 (local address 127.0.97.2:33285)
0620 03:03:49.489883 (+ 3350us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:49.489891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:49.489920 (+ 29us) server_negotiation.cc:407] Connection header received
0620 03:03:49.490018 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:49.490036 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:49.490227 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:49.490891 (+ 664us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:49.492175 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.493797 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.496717 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.498259 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.500351 (+ 2092us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:49.500910 (+ 559us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:49.500973 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 03:03:49.501139 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3178,"thread_start_us":3056,"threads_started":1}
I20260620 03:03:49.503563 19540 tablet_service.cc:1467] Processing CreateTablet for tablet dc79d812ee53452880f90178d78e9de3 (DEFAULT_TABLE table=test-table [id=f4eb785da8aa4e04960db05da780d4b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:03:49.503564 19279 tablet_service.cc:1467] Processing CreateTablet for tablet dc79d812ee53452880f90178d78e9de3 (DEFAULT_TABLE table=test-table [id=f4eb785da8aa4e04960db05da780d4b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:03:49.505105 19540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc79d812ee53452880f90178d78e9de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:49.506464 19410 tablet_service.cc:1467] Processing CreateTablet for tablet dc79d812ee53452880f90178d78e9de3 (DEFAULT_TABLE table=test-table [id=f4eb785da8aa4e04960db05da780d4b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:03:49.507517 19410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc79d812ee53452880f90178d78e9de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:49.507622 19279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc79d812ee53452880f90178d78e9de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:49.519804 19623 tablet_bootstrap.cc:492] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565: Bootstrap starting.
I20260620 03:03:49.522660 19625 tablet_bootstrap.cc:492] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0: Bootstrap starting.
I20260620 03:03:49.524374 19624 tablet_bootstrap.cc:492] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22: Bootstrap starting.
I20260620 03:03:49.526501 19625 tablet_bootstrap.cc:654] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:49.526489 19623 tablet_bootstrap.cc:654] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:49.527786 19624 tablet_bootstrap.cc:654] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:49.527926 19623 log.cc:824] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:49.528898 19625 log.cc:824] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:49.529314 19624 log.cc:824] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:49.534914 19624 tablet_bootstrap.cc:492] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22: No bootstrap required, opened a new log
I20260620 03:03:49.535326 19624 ts_tablet_manager.cc:1397] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 03:03:49.542992 19625 tablet_bootstrap.cc:492] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0: No bootstrap required, opened a new log
I20260620 03:03:49.542992 19623 tablet_bootstrap.cc:492] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565: No bootstrap required, opened a new log
I20260620 03:03:49.543272 19623 ts_tablet_manager.cc:1397] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.000s
I20260620 03:03:49.543269 19625 ts_tablet_manager.cc:1397] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.005s
I20260620 03:03:49.545233 19624 raft_consensus.cc:348] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d9b3c01a94ef6816deb046082f565" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33285 } } peers { permanent_uuid: "9d8cd812c6204190852cabe030585b22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40019 } } peers { permanent_uuid: "7e6d593f87074c02b55aa2d394bcf5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } }
I20260620 03:03:49.545617 19624 raft_consensus.cc:374] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:49.545758 19624 raft_consensus.cc:729] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d8cd812c6204190852cabe030585b22, State: Initialized, Role: FOLLOWER
I20260620 03:03:49.546375 19624 consensus_queue.cc:260] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d9b3c01a94ef6816deb046082f565" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33285 } } peers { permanent_uuid: "9d8cd812c6204190852cabe030585b22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40019 } } peers { permanent_uuid: "7e6d593f87074c02b55aa2d394bcf5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } }
I20260620 03:03:49.547416 19605 heartbeater.cc:503] Master 127.0.97.62:40143 was elected leader, sending a full tablet report...
I20260620 03:03:49.548157 19624 ts_tablet_manager.cc:1428] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
I20260620 03:03:49.549780 19344 heartbeater.cc:503] Master 127.0.97.62:40143 was elected leader, sending a full tablet report...
I20260620 03:03:49.553949 19625 raft_consensus.cc:348] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d9b3c01a94ef6816deb046082f565" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33285 } } peers { permanent_uuid: "9d8cd812c6204190852cabe030585b22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40019 } } peers { permanent_uuid: "7e6d593f87074c02b55aa2d394bcf5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } }
I20260620 03:03:49.554384 19625 raft_consensus.cc:374] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:49.554507 19625 raft_consensus.cc:729] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e6d593f87074c02b55aa2d394bcf5a0, State: Initialized, Role: FOLLOWER
I20260620 03:03:49.555153 19625 consensus_queue.cc:260] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d9b3c01a94ef6816deb046082f565" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33285 } } peers { permanent_uuid: "9d8cd812c6204190852cabe030585b22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40019 } } peers { permanent_uuid: "7e6d593f87074c02b55aa2d394bcf5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } }
I20260620 03:03:49.555292 19623 raft_consensus.cc:348] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d9b3c01a94ef6816deb046082f565" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33285 } } peers { permanent_uuid: "9d8cd812c6204190852cabe030585b22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40019 } } peers { permanent_uuid: "7e6d593f87074c02b55aa2d394bcf5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } }
I20260620 03:03:49.555768 19623 raft_consensus.cc:374] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:49.555876 19623 raft_consensus.cc:729] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd6d9b3c01a94ef6816deb046082f565, State: Initialized, Role: FOLLOWER
I20260620 03:03:49.556466 19623 consensus_queue.cc:260] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d9b3c01a94ef6816deb046082f565" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33285 } } peers { permanent_uuid: "9d8cd812c6204190852cabe030585b22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40019 } } peers { permanent_uuid: "7e6d593f87074c02b55aa2d394bcf5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } }
I20260620 03:03:49.557047 19625 ts_tablet_manager.cc:1428] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260620 03:03:49.557633 19475 heartbeater.cc:503] Master 127.0.97.62:40143 was elected leader, sending a full tablet report...
I20260620 03:03:49.558212 19623 ts_tablet_manager.cc:1428] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260620 03:03:49.659126 19631 raft_consensus.cc:482] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:03:49.659502 19631 raft_consensus.cc:504] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d9b3c01a94ef6816deb046082f565" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33285 } } peers { permanent_uuid: "9d8cd812c6204190852cabe030585b22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40019 } } peers { permanent_uuid: "7e6d593f87074c02b55aa2d394bcf5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } }
I20260620 03:03:49.661257 19631 leader_election.cc:288] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d8cd812c6204190852cabe030585b22 (127.0.97.3:40019), 7e6d593f87074c02b55aa2d394bcf5a0 (127.0.97.1:33685)
I20260620 03:03:49.668059 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:49.661798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56147 (local address 127.0.97.3:40019)
0620 03:03:49.661993 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:49.661998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:49.662520 (+ 522us) server_negotiation.cc:407] Connection header received
0620 03:03:49.662801 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:49.662808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:49.662889 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:49.662959 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:49.663636 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.664339 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.666424 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.667184 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.667331 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:49.667436 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:49.667700 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 03:03:49.667822 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:03:49.668229 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:49.661642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40019 (local address 127.0.97.2:56147)
0620 03:03:49.662401 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:49.662420 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:49.662570 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:49.663124 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:49.663127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:49.663148 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:49.663496 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:49.663504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.664469 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.664477 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:49.666240 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:49.666255 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.667366 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.667373 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:49.667543 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:49.667567 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 03:03:49.667672 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 03:03:49.667802 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"spinlock_wait_cycles":17152,"thread_start_us":409,"threads_started":1}
I20260620 03:03:49.668478 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:49.662807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52855 (local address 127.0.97.1:33685)
0620 03:03:49.663031 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:49.663037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:49.663192 (+ 155us) server_negotiation.cc:407] Connection header received
0620 03:03:49.663432 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:49.663437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:49.663519 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:49.663616 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:49.664134 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.664886 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.666783 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.667546 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.667678 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:49.667774 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:49.668080 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 03:03:49.668204 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 03:03:49.668570 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:49.662482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33685 (local address 127.0.97.2:52855)
0620 03:03:49.663078 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:49.663097 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:49.663281 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:49.663642 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:49.663646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:49.663669 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:49.663987 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:49.663995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.665016 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.665021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:49.666603 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:49.666615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.667745 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.667751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:49.667904 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:49.667926 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 03:03:49.668067 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 03:03:49.668204 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":290,"threads_started":1}
I20260620 03:03:49.669562 19299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc79d812ee53452880f90178d78e9de3" candidate_uuid: "cd6d9b3c01a94ef6816deb046082f565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e6d593f87074c02b55aa2d394bcf5a0" is_pre_election: true
I20260620 03:03:49.670147 19299 raft_consensus.cc:2454] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd6d9b3c01a94ef6816deb046082f565 in term 0.
I20260620 03:03:49.670099 19560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc79d812ee53452880f90178d78e9de3" candidate_uuid: "cd6d9b3c01a94ef6816deb046082f565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d8cd812c6204190852cabe030585b22" is_pre_election: true
I20260620 03:03:49.670603 19560 raft_consensus.cc:2454] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd6d9b3c01a94ef6816deb046082f565 in term 0.
I20260620 03:03:49.670861 19366 leader_election.cc:302] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e6d593f87074c02b55aa2d394bcf5a0, cd6d9b3c01a94ef6816deb046082f565; no voters:
I20260620 03:03:49.671247 19631 raft_consensus.cc:2781] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:03:49.671399 19631 raft_consensus.cc:482] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:03:49.671510 19631 raft_consensus.cc:3037] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:49.673727 19631 raft_consensus.cc:504] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d9b3c01a94ef6816deb046082f565" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33285 } } peers { permanent_uuid: "9d8cd812c6204190852cabe030585b22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40019 } } peers { permanent_uuid: "7e6d593f87074c02b55aa2d394bcf5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } }
I20260620 03:03:49.674371 19631 leader_election.cc:288] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [CANDIDATE]: Term 1 election: Requested vote from peers 9d8cd812c6204190852cabe030585b22 (127.0.97.3:40019), 7e6d593f87074c02b55aa2d394bcf5a0 (127.0.97.1:33685)
I20260620 03:03:49.674746 19560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc79d812ee53452880f90178d78e9de3" candidate_uuid: "cd6d9b3c01a94ef6816deb046082f565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d8cd812c6204190852cabe030585b22"
I20260620 03:03:49.674829 19299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc79d812ee53452880f90178d78e9de3" candidate_uuid: "cd6d9b3c01a94ef6816deb046082f565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e6d593f87074c02b55aa2d394bcf5a0"
I20260620 03:03:49.674938 19560 raft_consensus.cc:3037] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:49.675082 19299 raft_consensus.cc:3037] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:49.677259 19560 raft_consensus.cc:2454] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd6d9b3c01a94ef6816deb046082f565 in term 1.
I20260620 03:03:49.677326 19299 raft_consensus.cc:2454] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd6d9b3c01a94ef6816deb046082f565 in term 1.
I20260620 03:03:49.677701 19366 leader_election.cc:302] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d8cd812c6204190852cabe030585b22, cd6d9b3c01a94ef6816deb046082f565; no voters:
I20260620 03:03:49.677997 19631 raft_consensus.cc:2781] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:03:49.678496 19631 raft_consensus.cc:686] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [term 1 LEADER]: Becoming Leader. State: Replica: cd6d9b3c01a94ef6816deb046082f565, State: Running, Role: LEADER
I20260620 03:03:49.678984 19631 consensus_queue.cc:237] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d9b3c01a94ef6816deb046082f565" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33285 } } peers { permanent_uuid: "9d8cd812c6204190852cabe030585b22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40019 } } peers { permanent_uuid: "7e6d593f87074c02b55aa2d394bcf5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } }
I20260620 03:03:49.683907 19157 catalog_manager.cc:5495] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd6d9b3c01a94ef6816deb046082f565 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "cd6d9b3c01a94ef6816deb046082f565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d9b3c01a94ef6816deb046082f565" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d8cd812c6204190852cabe030585b22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e6d593f87074c02b55aa2d394bcf5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } health_report { overall_health: UNKNOWN } } }
W20260620 03:03:49.684410 19606 tablet.cc:2369] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:03:49.737376 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:49.731847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58264 (local address 127.0.97.2:33285)
0620 03:03:49.732070 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:49.732077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:49.732163 (+ 86us) server_negotiation.cc:407] Connection header received
0620 03:03:49.732404 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:49.732408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:49.732501 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:49.732663 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:03:49.733041 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.733743 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.735142 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.735547 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.735684 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:49.736305 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:03:49.736799 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:03:49.737002 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:49.737094 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 03:03:49.737135 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 03:03:49.737870 19476 tablet.cc:2369] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:03:49.745616 19560 raft_consensus.cc:1261] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22 [term 1 FOLLOWER]: Refusing update from remote peer cd6d9b3c01a94ef6816deb046082f565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:03:49.745640 19299 raft_consensus.cc:1261] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0 [term 1 FOLLOWER]: Refusing update from remote peer cd6d9b3c01a94ef6816deb046082f565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:03:49.746585 19634 consensus_queue.cc:1035] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d8cd812c6204190852cabe030585b22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:03:49.746903 19631 consensus_queue.cc:1035] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d593f87074c02b55aa2d394bcf5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:03:49.770946 19158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:03:49.771209 19158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:03:49.774755 19158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e17cb505ffb4b2b83ddbd7916e6f111: Sending DeleteTablet for 3 replicas of tablet dc79d812ee53452880f90178d78e9de3
I20260620 03:03:49.776047 19279 tablet_service.cc:1514] Processing DeleteTablet for tablet dc79d812ee53452880f90178d78e9de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:03:49 UTC) from {username='slave'} at 127.0.0.1:44330
I20260620 03:03:49.776077 19410 tablet_service.cc:1514] Processing DeleteTablet for tablet dc79d812ee53452880f90178d78e9de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:03:49 UTC) from {username='slave'} at 127.0.0.1:58256
I20260620 03:03:49.776149 19540 tablet_service.cc:1514] Processing DeleteTablet for tablet dc79d812ee53452880f90178d78e9de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:03:49 UTC) from {username='slave'} at 127.0.0.1:60564
I20260620 03:03:49.777724 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 19217
I20260620 03:03:49.789542 19647 tablet_replica.cc:330] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565: stopping tablet replica
I20260620 03:03:49.789803 19649 tablet_replica.cc:330] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0: stopping tablet replica
I20260620 03:03:49.790110 19647 raft_consensus.cc:2229] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:03:49.790374 19649 raft_consensus.cc:2229] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:03:49.790845 19647 raft_consensus.cc:2258] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:03:49.790942 19649 raft_consensus.cc:2258] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:03:49.793387 19647 ts_tablet_manager.cc:1905] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:03:49.793449 19649 ts_tablet_manager.cc:1905] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:03:49.800585 19649 ts_tablet_manager.cc:1918] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:03:49.800834 19649 log.cc:1196] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/dc79d812ee53452880f90178d78e9de3
I20260620 03:03:49.801434 19649 ts_tablet_manager.cc:1939] T dc79d812ee53452880f90178d78e9de3 P 7e6d593f87074c02b55aa2d394bcf5a0: Deleting consensus metadata
I20260620 03:03:49.802937 19647 ts_tablet_manager.cc:1918] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:03:49.803128 19647 log.cc:1196] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/dc79d812ee53452880f90178d78e9de3
I20260620 03:03:49.803721 19647 ts_tablet_manager.cc:1939] T dc79d812ee53452880f90178d78e9de3 P cd6d9b3c01a94ef6816deb046082f565: Deleting consensus metadata
I20260620 03:03:49.803676 19145 catalog_manager.cc:4841] TS 7e6d593f87074c02b55aa2d394bcf5a0 (127.0.97.1:33685): tablet dc79d812ee53452880f90178d78e9de3 (table test-table [id=f4eb785da8aa4e04960db05da780d4b7]) successfully deleted
I20260620 03:03:49.805517 19144 catalog_manager.cc:4841] TS cd6d9b3c01a94ef6816deb046082f565 (127.0.97.2:33285): tablet dc79d812ee53452880f90178d78e9de3 (table test-table [id=f4eb785da8aa4e04960db05da780d4b7]) successfully deleted
I20260620 03:03:49.805760 19648 tablet_replica.cc:330] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22: stopping tablet replica
I20260620 03:03:49.806131 19648 raft_consensus.cc:2229] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:03:49.806509 19648 raft_consensus.cc:2258] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:03:49.808218 19648 ts_tablet_manager.cc:1905] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:03:49.812099 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:49.779702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44344 (local address 127.0.97.1:33685)
0620 03:03:49.779910 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:49.779917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:49.804073 (+ 24156us) server_negotiation.cc:407] Connection header received
0620 03:03:49.804551 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:49.804557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:49.804654 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:49.804758 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:49.805379 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.806556 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.808540 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.809151 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.809276 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:49.810515 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:49.810527 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:49.810544 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:49.811273 (+ 729us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:49.811327 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:49.811350 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:49.811471 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:49.811638 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 03:03:49.811743 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 03:03:49.812727 19339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:49.815877 19648 ts_tablet_manager.cc:1918] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:03:49.816038 19648 log.cc:1196] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/dc79d812ee53452880f90178d78e9de3
I20260620 03:03:49.816599 19648 ts_tablet_manager.cc:1939] T dc79d812ee53452880f90178d78e9de3 P 9d8cd812c6204190852cabe030585b22: Deleting consensus metadata
I20260620 03:03:49.818315 19145 catalog_manager.cc:4841] TS 9d8cd812c6204190852cabe030585b22 (127.0.97.3:40019): tablet dc79d812ee53452880f90178d78e9de3 (table test-table [id=f4eb785da8aa4e04960db05da780d4b7]) successfully deleted
I20260620 03:03:49.906701 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 19217
I20260620 03:03:49.925108 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 19348
I20260620 03:03:49.932436 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:49.926331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58276 (local address 127.0.97.2:33285)
0620 03:03:49.926635 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:49.926639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:49.926655 (+ 16us) server_negotiation.cc:407] Connection header received
0620 03:03:49.926706 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:49.926710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:49.926782 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:49.926906 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:49.927325 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.928077 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.929337 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:49.929841 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:49.929979 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:49.931065 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:49.931079 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:49.931093 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:49.931693 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:49.931741 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:49.931761 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:49.931875 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:49.932010 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 03:03:49.932110 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260620 03:03:49.933266 19470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:50.048460 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 19348
I20260620 03:03:50.065011 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 19478
I20260620 03:03:50.071899 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:50.065798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.0.97.3:40019)
0620 03:03:50.066043 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:50.066048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:50.066070 (+ 22us) server_negotiation.cc:407] Connection header received
0620 03:03:50.066242 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:50.066246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:50.066327 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:50.066418 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:50.066755 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:50.067428 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:50.068641 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:50.069024 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:50.069114 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:50.070375 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:50.070390 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:50.070413 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:50.071212 (+ 799us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:50.071258 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:50.071277 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:50.071377 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:50.071474 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 03:03:50.071555 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 03:03:50.072517 19600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:50.164723 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 19478
I20260620 03:03:50.181699 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 19127
I20260620 03:03:50.182792 19188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:50.291762 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 19127
2026-06-20T03:03:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2883 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:03:50 dist-test-slave-f1j0 krb5kdc[19657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:03:50 dist-test-slave-f1j0 krb5kdc[19657](info): set up 2 sockets
Jun 20 03:03:50 dist-test-slave-f1j0 krb5kdc[19657](info): commencing operation
krb5kdc: starting...
W20260620 03:03:51.401873 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:03:51 dist-test-slave-f1j0 krb5kdc[19657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:03:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:03:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:03:51.649464 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36677
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:03:51.975139 19670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:51.975445 19670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:51.975534 19670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:51.982713 19670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:03:51.982820 19670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:51.982865 19670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:51.982925 19670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:03:51.982969 19670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:03:51.992753 19670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:51.994160 19670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:51.995410 19670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:52.007114 19675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:52.007197 19676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:52.007157 19678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:52.007956 19670 server_base.cc:1034] running on GCE node
I20260620 03:03:52.008687 19670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:52.010147 19670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:52.011346 19670 hybrid_clock.cc:648] HybridClock initialized: now 1781924632011307 us; error 30 us; skew 500 ppm
I20260620 03:03:52.011703 19670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:03:52 dist-test-slave-f1j0 krb5kdc[19657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:03:52.014989 19670 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:03:52.016484 19670 webserver.cc:460] Webserver started at http://127.0.97.62:41757/ using document root <none> and password file <none>
I20260620 03:03:52.017176 19670 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:52.017313 19670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:52.017699 19670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:52.020623 19670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "fc0b86edce594b43bc7169231360b03b"
format_stamp: "Formatted at 2026-06-20 03:03:52 on dist-test-slave-f1j0"
server_key: "7e6e95d39bb1fd35c257223b4960c291"
server_key_iv: "ec02a121abe650062efd563a49a5db61"
server_key_version: "encryptionkey@0"
I20260620 03:03:52.021355 19670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "fc0b86edce594b43bc7169231360b03b"
format_stamp: "Formatted at 2026-06-20 03:03:52 on dist-test-slave-f1j0"
server_key: "7e6e95d39bb1fd35c257223b4960c291"
server_key_iv: "ec02a121abe650062efd563a49a5db61"
server_key_version: "encryptionkey@0"
I20260620 03:03:52.026435 19670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260620 03:03:52.029810 19685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:52.031118 19670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:03:52.031292 19670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "fc0b86edce594b43bc7169231360b03b"
format_stamp: "Formatted at 2026-06-20 03:03:52 on dist-test-slave-f1j0"
server_key: "7e6e95d39bb1fd35c257223b4960c291"
server_key_iv: "ec02a121abe650062efd563a49a5db61"
server_key_version: "encryptionkey@0"
I20260620 03:03:52.031482 19670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:52.046216 19670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:52.050489 19670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:52.050773 19670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:52.068753 19670 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36677
I20260620 03:03:52.068789 19736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36677 every 8 connection(s)
I20260620 03:03:52.070344 19670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:03:52.072326 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 19670
I20260620 03:03:52.072504 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:03:52.073091 388 external_mini_cluster.cc:1428] Setting key 5444bff9b19bd71fe87d0811634ae8bb
I20260620 03:03:52.074745 19737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:52.085731 19737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b: Bootstrap starting.
Jun 20 03:03:52 dist-test-slave-f1j0 krb5kdc[19657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:03:52.089921 19737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:52.091024 19737 log.cc:824] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:52.094331 19737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b: No bootstrap required, opened a new log
I20260620 03:03:52.095502 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:52.074997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58404 (local address 127.0.97.62:36677)
0620 03:03:52.075759 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:52.075771 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:52.075812 (+ 41us) server_negotiation.cc:407] Connection header received
0620 03:03:52.077429 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:52.077449 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:52.077904 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:52.078302 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:52.079719 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:52.081022 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:52.082400 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:52.083090 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:52.083289 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:52.087933 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:52.087947 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:52.087971 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:52.088024 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:52.091627 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:52.092317 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:52.092323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:52.092329 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:52.092491 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:52.092906 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:52.092909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:52.092911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:52.093417 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:52.093708 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:52.094119 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 03:03:52.094491 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":325,"threads_started":1}
I20260620 03:03:52.102210 19737 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0b86edce594b43bc7169231360b03b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36677 } }
I20260620 03:03:52.102612 19737 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:52.102725 19737 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc0b86edce594b43bc7169231360b03b, State: Initialized, Role: FOLLOWER
I20260620 03:03:52.103276 19737 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0b86edce594b43bc7169231360b03b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36677 } }
I20260620 03:03:52.103502 19737 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:03:52.103670 19737 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:03:52.103843 19737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:52.106236 19737 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0b86edce594b43bc7169231360b03b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36677 } }
I20260620 03:03:52.106745 19737 leader_election.cc:302] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc0b86edce594b43bc7169231360b03b; no voters:
I20260620 03:03:52.107477 19737 leader_election.cc:288] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:03:52.107681 19742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:03:52.108393 19742 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b [term 1 LEADER]: Becoming Leader. State: Replica: fc0b86edce594b43bc7169231360b03b, State: Running, Role: LEADER
I20260620 03:03:52.108974 19742 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0b86edce594b43bc7169231360b03b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36677 } }
I20260620 03:03:52.109694 19737 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:03:52.112134 19743 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc0b86edce594b43bc7169231360b03b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0b86edce594b43bc7169231360b03b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36677 } } }
I20260620 03:03:52.112618 19743 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b [sys.catalog]: This master's current role is: LEADER
I20260620 03:03:52.114145 19744 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc0b86edce594b43bc7169231360b03b. Latest consensus state: current_term: 1 leader_uuid: "fc0b86edce594b43bc7169231360b03b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0b86edce594b43bc7169231360b03b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36677 } } }
I20260620 03:03:52.114584 19744 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b [sys.catalog]: This master's current role is: LEADER
I20260620 03:03:52.125260 19750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:03:52.131870 19750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:03:52.141417 19750 catalog_manager.cc:1347] Generated new cluster ID: 1f915910fc7d45fd8ba4e5883089c861
I20260620 03:03:52.141546 19750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:03:52.169857 19750 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:03:52.171806 19750 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:03:52.187840 19750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b: Generated new TSK 0
I20260620 03:03:52.188974 19750 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 03:03:52.253000 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36677
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:03:52.584928 19763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:52.585255 19763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:52.585425 19763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:52.592813 19763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:52.592955 19763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:52.593137 19763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:03:52.602895 19763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:52.604547 19763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:52.605954 19763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:52.619294 19768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:52.619273 19771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:52.619352 19769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:52.620458 19763 server_base.cc:1034] running on GCE node
I20260620 03:03:52.621122 19763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:52.622076 19763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:52.623330 19763 hybrid_clock.cc:648] HybridClock initialized: now 1781924632623293 us; error 48 us; skew 500 ppm
I20260620 03:03:52.623689 19763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:03:52 dist-test-slave-f1j0 krb5kdc[19657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:03:52.626933 19763 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 03:03:52.628259 19763 webserver.cc:460] Webserver started at http://127.0.97.1:37459/ using document root <none> and password file <none>
I20260620 03:03:52.628904 19763 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:52.628998 19763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:52.629249 19763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:52.632085 19763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "09ed1345bb1949469dbb614f0f4e59c5"
format_stamp: "Formatted at 2026-06-20 03:03:52 on dist-test-slave-f1j0"
server_key: "ec324e4109933c7f0e603d95bcac1711"
server_key_iv: "bbf8f097794238c94396bac8ad6c2247"
server_key_version: "encryptionkey@0"
I20260620 03:03:52.632807 19763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "09ed1345bb1949469dbb614f0f4e59c5"
format_stamp: "Formatted at 2026-06-20 03:03:52 on dist-test-slave-f1j0"
server_key: "ec324e4109933c7f0e603d95bcac1711"
server_key_iv: "bbf8f097794238c94396bac8ad6c2247"
server_key_version: "encryptionkey@0"
I20260620 03:03:52.637996 19763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 03:03:52.641209 19778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:52.642526 19763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:03:52.642655 19763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "09ed1345bb1949469dbb614f0f4e59c5"
format_stamp: "Formatted at 2026-06-20 03:03:52 on dist-test-slave-f1j0"
server_key: "ec324e4109933c7f0e603d95bcac1711"
server_key_iv: "bbf8f097794238c94396bac8ad6c2247"
server_key_version: "encryptionkey@0"
I20260620 03:03:52.642819 19763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:52.653743 19763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:52.657730 19763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:52.657960 19763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:52.658802 19763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:52.660007 19763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:52.660076 19763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:52.660157 19763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:52.660202 19763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:52.692853 19763 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45373
I20260620 03:03:52.692899 19890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45373 every 8 connection(s)
I20260620 03:03:52.694271 19763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:03:52.699992 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 19763
I20260620 03:03:52.700189 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:03:52.700683 388 external_mini_cluster.cc:1428] Setting key c618646b23b91655244a17bf96863d3b
Jun 20 03:03:52 dist-test-slave-f1j0 krb5kdc[19657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:03:52.714700 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:52.696912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42615 (local address 127.0.97.62:36677)
0620 03:03:52.697346 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:52.697350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:52.698392 (+ 1042us) server_negotiation.cc:407] Connection header received
0620 03:03:52.700311 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:52.700317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:52.700415 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:52.700548 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:52.702255 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:52.703056 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:52.704616 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:52.705045 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:52.705190 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:52.708690 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:52.708703 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:52.708707 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:52.708739 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:52.711638 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:52.712350 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:52.712355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:52.712359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:52.712480 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:52.712901 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:52.712906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:52.712909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:52.713182 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:52.713344 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:52.714097 (+ 753us) server_negotiation.cc:299] Negotiation successful
0620 03:03:52.714329 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":222,"threads_started":1}
I20260620 03:03:52.715781 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:52.697473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36677 (local address 127.0.97.1:42615)
0620 03:03:52.698227 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:52.698273 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:52.699998 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:52.700809 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:52.700822 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:52.701356 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:03:52.702087 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:52.702105 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:52.703216 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:52.703220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:52.704457 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:52.704469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:52.705278 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:52.705287 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:52.705534 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:52.706308 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:03:52.706330 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:03:52.708464 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:03:52.711775 (+ 3311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:03:52.711784 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:03:52.711801 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:03:52.712223 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:03:52.712590 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:03:52.712595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:03:52.712598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:03:52.712781 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:03:52.713317 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:03:52.713325 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:03:52.713717 (+ 392us) client_negotiation.cc:769] Sending connection context
0620 03:03:52.713986 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 03:03:52.714531 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":279,"threads_started":1}
I20260620 03:03:52.717548 19891 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36677
I20260620 03:03:52.717978 19891 heartbeater.cc:464] Registering TS with master...
I20260620 03:03:52.718734 19891 heartbeater.cc:511] Master 127.0.97.62:36677 requested a full tablet report, sending...
I20260620 03:03:52.721030 19702 ts_manager.cc:194] Registered new tserver with Master: 09ed1345bb1949469dbb614f0f4e59c5 (127.0.97.1:45373)
I20260620 03:03:52.722577 19702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42615
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 03:03:52.760530 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36677
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:03:53.083168 19897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:53.083431 19897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:53.083593 19897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:53.091859 19897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:53.092001 19897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:53.092202 19897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:03:53.102489 19897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:53.103842 19897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:53.105109 19897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:53.118729 19905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:53.118700 19902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:53.118793 19903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:53.119648 19897 server_base.cc:1034] running on GCE node
I20260620 03:03:53.120275 19897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:53.121320 19897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:53.122524 19897 hybrid_clock.cc:648] HybridClock initialized: now 1781924633122463 us; error 44 us; skew 500 ppm
I20260620 03:03:53.122876 19897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:03:53 dist-test-slave-f1j0 krb5kdc[19657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:03:53.126410 19897 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 03:03:53.127879 19897 webserver.cc:460] Webserver started at http://127.0.97.2:36401/ using document root <none> and password file <none>
I20260620 03:03:53.128561 19897 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:53.128661 19897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:53.128919 19897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:53.131938 19897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "611e720a917f4f0096f7a646622bd880"
format_stamp: "Formatted at 2026-06-20 03:03:53 on dist-test-slave-f1j0"
server_key: "0ef18a3e96cc4da7e9e08911d7604c10"
server_key_iv: "73cdb2031c7c07b58c47fbb9e0d6b067"
server_key_version: "encryptionkey@0"
I20260620 03:03:53.132733 19897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "611e720a917f4f0096f7a646622bd880"
format_stamp: "Formatted at 2026-06-20 03:03:53 on dist-test-slave-f1j0"
server_key: "0ef18a3e96cc4da7e9e08911d7604c10"
server_key_iv: "73cdb2031c7c07b58c47fbb9e0d6b067"
server_key_version: "encryptionkey@0"
I20260620 03:03:53.138514 19897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 03:03:53.142048 19912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:53.143409 19897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 03:03:53.143596 19897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "611e720a917f4f0096f7a646622bd880"
format_stamp: "Formatted at 2026-06-20 03:03:53 on dist-test-slave-f1j0"
server_key: "0ef18a3e96cc4da7e9e08911d7604c10"
server_key_iv: "73cdb2031c7c07b58c47fbb9e0d6b067"
server_key_version: "encryptionkey@0"
I20260620 03:03:53.143807 19897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:53.161741 19897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:53.165858 19897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:53.166163 19897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:53.167135 19897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:53.168503 19897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:53.168612 19897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:53.168772 19897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:53.168845 19897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:53.201963 19897 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44261
I20260620 03:03:53.202008 20024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44261 every 8 connection(s)
I20260620 03:03:53.203299 19897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:03:53.205269 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 19897
I20260620 03:03:53.205471 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:03:53.206022 388 external_mini_cluster.cc:1428] Setting key 24dba014bce6678dc3caa33bfd4a663a
Jun 20 03:03:53 dist-test-slave-f1j0 krb5kdc[19657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:03:53.224077 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:53.206170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45987 (local address 127.0.97.62:36677)
0620 03:03:53.206390 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:53.206396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:53.207820 (+ 1424us) server_negotiation.cc:407] Connection header received
0620 03:03:53.209838 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:53.209843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:53.209954 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:53.210063 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:53.211599 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.212299 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.213875 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.214311 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.214448 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:53.218131 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:53.218136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:53.218139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:53.218171 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:53.220929 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:53.221730 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:53.221735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:53.221739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:53.221875 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:53.222317 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:53.222322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:53.222325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:53.222611 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:53.222767 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:53.223482 (+ 715us) server_negotiation.cc:299] Negotiation successful
0620 03:03:53.223731 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 03:03:53.225138 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:53.206906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36677 (local address 127.0.97.2:45987)
0620 03:03:53.207677 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:53.207714 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:53.209508 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:53.210313 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:53.210321 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:53.210830 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:03:53.211434 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:53.211450 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.212442 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.212450 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:53.213737 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:53.213749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.214538 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.214545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:53.214785 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:53.215482 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:03:53.215508 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:03:53.217847 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:03:53.221086 (+ 3239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:03:53.221095 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:03:53.221113 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:03:53.221543 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:03:53.221988 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:03:53.221994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:03:53.221998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:03:53.222196 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:03:53.222737 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:03:53.222745 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:03:53.223097 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 03:03:53.223368 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 03:03:53.223930 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":324,"threads_started":1}
I20260620 03:03:53.226683 20025 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36677
I20260620 03:03:53.227027 20025 heartbeater.cc:464] Registering TS with master...
I20260620 03:03:53.227977 20025 heartbeater.cc:511] Master 127.0.97.62:36677 requested a full tablet report, sending...
I20260620 03:03:53.229808 19702 ts_manager.cc:194] Registered new tserver with Master: 611e720a917f4f0096f7a646622bd880 (127.0.97.2:44261)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 03:03:53.230741 19702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 03:03:53.262233 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36677
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:03:53.587007 20030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:53.587277 20030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:53.587441 20030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:53.594512 20030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:53.594619 20030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:53.594828 20030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:03:53.604292 20030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:53.605643 20030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:53.606958 20030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:53.620407 20036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:53.620429 20038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:53.620625 20035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:53.621410 20030 server_base.cc:1034] running on GCE node
I20260620 03:03:53.622089 20030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:53.623070 20030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:53.624358 20030 hybrid_clock.cc:648] HybridClock initialized: now 1781924633624281 us; error 58 us; skew 500 ppm
I20260620 03:03:53.624748 20030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:03:53 dist-test-slave-f1j0 krb5kdc[19657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:03:53.628015 20030 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 03:03:53.629601 20030 webserver.cc:460] Webserver started at http://127.0.97.3:38449/ using document root <none> and password file <none>
I20260620 03:03:53.630301 20030 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:53.630401 20030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:53.630669 20030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:53.633701 20030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "6f2b5808bd8d45ff847a4971f66205d9"
format_stamp: "Formatted at 2026-06-20 03:03:53 on dist-test-slave-f1j0"
server_key: "c826efe678ac679e521366d8777a4bc8"
server_key_iv: "b4ddbec4dba1ce6bfa34771fef24996c"
server_key_version: "encryptionkey@0"
I20260620 03:03:53.634624 20030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6f2b5808bd8d45ff847a4971f66205d9"
format_stamp: "Formatted at 2026-06-20 03:03:53 on dist-test-slave-f1j0"
server_key: "c826efe678ac679e521366d8777a4bc8"
server_key_iv: "b4ddbec4dba1ce6bfa34771fef24996c"
server_key_version: "encryptionkey@0"
I20260620 03:03:53.639813 20030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:03:53.643463 20045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:53.644975 20030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:03:53.645161 20030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "6f2b5808bd8d45ff847a4971f66205d9"
format_stamp: "Formatted at 2026-06-20 03:03:53 on dist-test-slave-f1j0"
server_key: "c826efe678ac679e521366d8777a4bc8"
server_key_iv: "b4ddbec4dba1ce6bfa34771fef24996c"
server_key_version: "encryptionkey@0"
I20260620 03:03:53.645370 20030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:53.659185 20030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:53.663102 20030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:53.663394 20030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:53.664348 20030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:53.665835 20030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:53.665906 20030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:53.666049 20030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:53.666111 20030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:53.699213 20030 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38037
I20260620 03:03:53.699298 20157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38037 every 8 connection(s)
I20260620 03:03:53.700418 20030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:03:53.705940 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 20030
I20260620 03:03:53.706143 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:03:53.706565 388 external_mini_cluster.cc:1428] Setting key e20cc5cc52864db478394cf25d5061e2
Jun 20 03:03:53 dist-test-slave-f1j0 krb5kdc[19657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:03:53.719970 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:53.702890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47071 (local address 127.0.97.62:36677)
0620 03:03:53.703110 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:53.703116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:53.704266 (+ 1150us) server_negotiation.cc:407] Connection header received
0620 03:03:53.706055 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:53.706060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:53.706152 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:53.706268 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:53.708078 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.708983 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.710603 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.710994 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.711132 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:53.714607 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:53.714612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:53.714616 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:53.714647 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:53.717183 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:53.718027 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:53.718031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:53.718033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:53.718158 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:53.718512 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:53.718515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:53.718517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:53.718773 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:53.718883 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:53.719501 (+ 618us) server_negotiation.cc:299] Negotiation successful
0620 03:03:53.719694 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 03:03:53.721159 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:53.703408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36677 (local address 127.0.97.3:47071)
0620 03:03:53.704110 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:53.704151 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:53.705794 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:53.706494 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:53.706502 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:53.707261 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:03:53.707903 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:53.707934 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.709133 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.709139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:53.710454 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:53.710465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.711213 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.711223 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:53.711457 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:53.712130 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:03:53.712153 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:03:53.714386 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:03:53.717379 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:03:53.717390 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:03:53.717411 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:03:53.717918 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:03:53.718249 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:03:53.718252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:03:53.718255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:03:53.718420 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:03:53.718876 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:03:53.718881 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:03:53.719177 (+ 296us) client_negotiation.cc:769] Sending connection context
0620 03:03:53.719386 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 03:03:53.719853 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":278,"threads_started":1}
I20260620 03:03:53.722440 20158 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36677
I20260620 03:03:53.722709 20158 heartbeater.cc:464] Registering TS with master...
I20260620 03:03:53.723351 20158 heartbeater.cc:511] Master 127.0.97.62:36677 requested a full tablet report, sending...
I20260620 03:03:53.724634 19702 ts_manager.cc:194] Registered new tserver with Master: 6f2b5808bd8d45ff847a4971f66205d9 (127.0.97.3:38037)
I20260620 03:03:53.725455 19702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47071
I20260620 03:03:53.725812 19891 heartbeater.cc:503] Master 127.0.97.62:36677 was elected leader, sending a full tablet report...
I20260620 03:03:53.733927 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:03:53.750932 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:53.738205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42036 (local address 127.0.97.62:36677)
0620 03:03:53.738367 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:53.738371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:53.738848 (+ 477us) server_negotiation.cc:407] Connection header received
0620 03:03:53.738995 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:53.738999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:53.739067 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:53.739148 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:53.740480 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.741172 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.742574 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.742978 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.743077 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:53.745437 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:53.745442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:53.745445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:53.745475 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:53.748354 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:53.749084 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:53.749088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:53.749090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:53.749204 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:53.749648 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:53.749651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:53.749652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:53.750026 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:53.750216 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:53.750541 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 03:03:53.750698 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 03:03:53.754689 19702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:03:53.761400 19702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:03:53.789284 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:53.773062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44261 (local address 127.0.0.1:60322)
0620 03:03:53.774954 (+ 1892us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:53.775003 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:53.775221 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:53.779285 (+ 4064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:53.779296 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:53.779339 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:53.779833 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:53.779849 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.782670 (+ 2821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.782678 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:53.785877 (+ 3199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:53.785901 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.788295 (+ 2394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.788300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:53.788561 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:53.788627 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 03:03:53.788748 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 03:03:53.788945 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1562,"thread_start_us":5308,"threads_started":1}
I20260620 03:03:53.789695 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:53.773279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60322 (local address 127.0.97.2:44261)
0620 03:03:53.778532 (+ 5253us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:53.778542 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:53.778571 (+ 29us) server_negotiation.cc:407] Connection header received
0620 03:03:53.778678 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:53.778685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:53.778985 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:53.780080 (+ 1095us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:53.780350 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.782470 (+ 2120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.786153 (+ 3683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.788069 (+ 1916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.788870 (+ 801us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:53.789090 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:53.789155 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 03:03:53.789316 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5081,"thread_start_us":4943,"threads_started":1}
I20260620 03:03:53.791607 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:53.773743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45373 (local address 127.0.0.1:47284)
0620 03:03:53.778598 (+ 4855us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:53.778612 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:53.778752 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:53.779729 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:53.779737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:53.779776 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:53.781609 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:53.781630 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.784151 (+ 2521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.784159 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:53.788317 (+ 4158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:53.788337 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.790851 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.790857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:53.791032 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:53.791065 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 03:03:53.791155 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 03:03:53.791278 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4755,"thread_start_us":981,"threads_started":1}
I20260620 03:03:53.791971 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:53.774517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47284 (local address 127.0.97.1:45373)
0620 03:03:53.778210 (+ 3693us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:53.778223 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:53.778880 (+ 657us) server_negotiation.cc:407] Connection header received
0620 03:03:53.778986 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:53.778995 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:53.779232 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:53.779430 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:53.781847 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.783940 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.788549 (+ 4609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.790616 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.790743 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:53.790985 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:53.791241 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 03:03:53.791399 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3445,"thread_start_us":3259,"threads_started":1}
I20260620 03:03:53.793864 19960 tablet_service.cc:1467] Processing CreateTablet for tablet 131cb55aeb46444d9fd49d41bad976bc (DEFAULT_TABLE table=test-table [id=3e950ff94d0040b3ab657a122a35b73d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:03:53.795281 19960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 131cb55aeb46444d9fd49d41bad976bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:53.795075 19826 tablet_service.cc:1467] Processing CreateTablet for tablet 131cb55aeb46444d9fd49d41bad976bc (DEFAULT_TABLE table=test-table [id=3e950ff94d0040b3ab657a122a35b73d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:03:53.796476 19826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 131cb55aeb46444d9fd49d41bad976bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:53.797333 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:53.778744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38037 (local address 127.0.0.1:32882)
0620 03:03:53.787999 (+ 9255us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:53.788028 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:53.788148 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:53.789894 (+ 1746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:53.789898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:53.789932 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:53.790262 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:53.790274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.792610 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.792616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:53.794540 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:53.794565 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.796501 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.796508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:53.796690 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:53.796728 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 03:03:53.796836 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 03:03:53.796971 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9119,"thread_start_us":9498,"threads_started":1}
I20260620 03:03:53.797376 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:53.779103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32882 (local address 127.0.97.3:38037)
0620 03:03:53.788866 (+ 9763us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:53.788875 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:53.788911 (+ 36us) server_negotiation.cc:407] Connection header received
0620 03:03:53.789008 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:53.789015 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:53.789269 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:53.790481 (+ 1212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:53.790747 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.792462 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.794713 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.796297 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.796438 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:53.796734 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:53.796880 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 03:03:53.797038 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9580,"thread_start_us":9452,"threads_started":1}
I20260620 03:03:53.801079 20093 tablet_service.cc:1467] Processing CreateTablet for tablet 131cb55aeb46444d9fd49d41bad976bc (DEFAULT_TABLE table=test-table [id=3e950ff94d0040b3ab657a122a35b73d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:03:53.802492 20093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 131cb55aeb46444d9fd49d41bad976bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:53.807900 20176 tablet_bootstrap.cc:492] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5: Bootstrap starting.
I20260620 03:03:53.809974 20177 tablet_bootstrap.cc:492] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880: Bootstrap starting.
I20260620 03:03:53.811908 20176 tablet_bootstrap.cc:654] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:53.813303 20177 tablet_bootstrap.cc:654] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:53.813272 20176 log.cc:824] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:53.814658 20177 log.cc:824] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:53.817200 20180 tablet_bootstrap.cc:492] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9: Bootstrap starting.
I20260620 03:03:53.817803 20177 tablet_bootstrap.cc:492] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880: No bootstrap required, opened a new log
I20260620 03:03:53.818081 20177 ts_tablet_manager.cc:1397] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260620 03:03:53.818094 20176 tablet_bootstrap.cc:492] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5: No bootstrap required, opened a new log
I20260620 03:03:53.818364 20176 ts_tablet_manager.cc:1397] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20260620 03:03:53.820741 20180 tablet_bootstrap.cc:654] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:53.821913 20180 log.cc:824] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:53.825551 20180 tablet_bootstrap.cc:492] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9: No bootstrap required, opened a new log
I20260620 03:03:53.825826 20180 ts_tablet_manager.cc:1397] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260620 03:03:53.826617 20176 raft_consensus.cc:348] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "611e720a917f4f0096f7a646622bd880" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44261 } } peers { permanent_uuid: "6f2b5808bd8d45ff847a4971f66205d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38037 } } peers { permanent_uuid: "09ed1345bb1949469dbb614f0f4e59c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45373 } }
I20260620 03:03:53.826965 20176 raft_consensus.cc:374] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:53.827072 20176 raft_consensus.cc:729] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ed1345bb1949469dbb614f0f4e59c5, State: Initialized, Role: FOLLOWER
I20260620 03:03:53.827687 20176 consensus_queue.cc:260] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "611e720a917f4f0096f7a646622bd880" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44261 } } peers { permanent_uuid: "6f2b5808bd8d45ff847a4971f66205d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38037 } } peers { permanent_uuid: "09ed1345bb1949469dbb614f0f4e59c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45373 } }
I20260620 03:03:53.828275 20177 raft_consensus.cc:348] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "611e720a917f4f0096f7a646622bd880" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44261 } } peers { permanent_uuid: "6f2b5808bd8d45ff847a4971f66205d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38037 } } peers { permanent_uuid: "09ed1345bb1949469dbb614f0f4e59c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45373 } }
I20260620 03:03:53.828704 20177 raft_consensus.cc:374] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:53.828785 20177 raft_consensus.cc:729] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 611e720a917f4f0096f7a646622bd880, State: Initialized, Role: FOLLOWER
I20260620 03:03:53.829404 20176 ts_tablet_manager.cc:1428] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260620 03:03:53.829447 20177 consensus_queue.cc:260] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "611e720a917f4f0096f7a646622bd880" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44261 } } peers { permanent_uuid: "6f2b5808bd8d45ff847a4971f66205d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38037 } } peers { permanent_uuid: "09ed1345bb1949469dbb614f0f4e59c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45373 } }
I20260620 03:03:53.838202 20025 heartbeater.cc:503] Master 127.0.97.62:36677 was elected leader, sending a full tablet report...
I20260620 03:03:53.838939 20177 ts_tablet_manager.cc:1428] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260620 03:03:53.838644 20180 raft_consensus.cc:348] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "611e720a917f4f0096f7a646622bd880" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44261 } } peers { permanent_uuid: "6f2b5808bd8d45ff847a4971f66205d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38037 } } peers { permanent_uuid: "09ed1345bb1949469dbb614f0f4e59c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45373 } }
I20260620 03:03:53.839087 20180 raft_consensus.cc:374] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:53.839185 20180 raft_consensus.cc:729] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f2b5808bd8d45ff847a4971f66205d9, State: Initialized, Role: FOLLOWER
I20260620 03:03:53.839747 20180 consensus_queue.cc:260] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "611e720a917f4f0096f7a646622bd880" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44261 } } peers { permanent_uuid: "6f2b5808bd8d45ff847a4971f66205d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38037 } } peers { permanent_uuid: "09ed1345bb1949469dbb614f0f4e59c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45373 } }
I20260620 03:03:53.840795 20158 heartbeater.cc:503] Master 127.0.97.62:36677 was elected leader, sending a full tablet report...
I20260620 03:03:53.841487 20180 ts_tablet_manager.cc:1428] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
W20260620 03:03:53.946645 19892 tablet.cc:2369] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:03:53.951987 20159 tablet.cc:2369] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 03:03:53.955654 20026 tablet.cc:2369] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:03:53.959012 20183 raft_consensus.cc:482] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:03:53.959280 20183 raft_consensus.cc:504] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "611e720a917f4f0096f7a646622bd880" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44261 } } peers { permanent_uuid: "6f2b5808bd8d45ff847a4971f66205d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38037 } } peers { permanent_uuid: "09ed1345bb1949469dbb614f0f4e59c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45373 } }
I20260620 03:03:53.960906 20183 leader_election.cc:288] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f2b5808bd8d45ff847a4971f66205d9 (127.0.97.3:38037), 09ed1345bb1949469dbb614f0f4e59c5 (127.0.97.1:45373)
I20260620 03:03:53.968106 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:53.961310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35187 (local address 127.0.97.1:45373)
0620 03:03:53.961511 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:53.961517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:53.962189 (+ 672us) server_negotiation.cc:407] Connection header received
0620 03:03:53.962505 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:53.962514 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:53.962597 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:53.962692 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:53.963248 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.964148 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.966148 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.967096 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.967199 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:53.967274 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:53.967703 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 03:03:53.967837 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 03:03:53.968113 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:53.961263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45373 (local address 127.0.97.2:35187)
0620 03:03:53.962081 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:53.962103 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:53.962326 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:53.962709 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:53.962714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:53.962740 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:53.963103 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:53.963121 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.964276 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.964282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:53.965990 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:53.966001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.967315 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.967322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:53.967493 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:53.967533 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 03:03:53.967658 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 03:03:53.967795 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":424,"threads_started":1}
I20260620 03:03:53.968571 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:53.961380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46481 (local address 127.0.97.3:38037)
0620 03:03:53.961589 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:53.961596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:53.962600 (+ 1004us) server_negotiation.cc:407] Connection header received
0620 03:03:53.962799 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:53.962805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:53.962885 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:53.962936 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:53.963750 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.964700 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.966636 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.967575 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.967688 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:53.967783 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:53.968157 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 03:03:53.968402 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 03:03:53.968750 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:53.961275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38037 (local address 127.0.97.2:46481)
0620 03:03:53.962459 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:53.962477 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:53.962625 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:53.963154 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:53.963159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:53.963189 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:53.963596 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:53.963609 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.964849 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.964853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:53.966472 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:53.966486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:53.967782 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:53.967788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:53.967972 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:53.968003 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 03:03:53.968110 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 03:03:53.968382 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"mutex_wait_us":30,"thread_start_us":845,"threads_started":1}
I20260620 03:03:53.969218 19846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "131cb55aeb46444d9fd49d41bad976bc" candidate_uuid: "611e720a917f4f0096f7a646622bd880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ed1345bb1949469dbb614f0f4e59c5" is_pre_election: true
I20260620 03:03:53.969830 19846 raft_consensus.cc:2454] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 611e720a917f4f0096f7a646622bd880 in term 0.
I20260620 03:03:53.969791 20113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "131cb55aeb46444d9fd49d41bad976bc" candidate_uuid: "611e720a917f4f0096f7a646622bd880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f2b5808bd8d45ff847a4971f66205d9" is_pre_election: true
I20260620 03:03:53.970281 20113 raft_consensus.cc:2454] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 611e720a917f4f0096f7a646622bd880 in term 0.
I20260620 03:03:53.970510 19913 leader_election.cc:302] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ed1345bb1949469dbb614f0f4e59c5, 611e720a917f4f0096f7a646622bd880; no voters:
I20260620 03:03:53.970932 20183 raft_consensus.cc:2781] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:03:53.971082 20183 raft_consensus.cc:482] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:03:53.971199 20183 raft_consensus.cc:3037] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:53.973264 20183 raft_consensus.cc:504] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "611e720a917f4f0096f7a646622bd880" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44261 } } peers { permanent_uuid: "6f2b5808bd8d45ff847a4971f66205d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38037 } } peers { permanent_uuid: "09ed1345bb1949469dbb614f0f4e59c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45373 } }
I20260620 03:03:53.973922 20183 leader_election.cc:288] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [CANDIDATE]: Term 1 election: Requested vote from peers 6f2b5808bd8d45ff847a4971f66205d9 (127.0.97.3:38037), 09ed1345bb1949469dbb614f0f4e59c5 (127.0.97.1:45373)
I20260620 03:03:53.974395 19846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "131cb55aeb46444d9fd49d41bad976bc" candidate_uuid: "611e720a917f4f0096f7a646622bd880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ed1345bb1949469dbb614f0f4e59c5"
I20260620 03:03:53.974368 20113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "131cb55aeb46444d9fd49d41bad976bc" candidate_uuid: "611e720a917f4f0096f7a646622bd880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f2b5808bd8d45ff847a4971f66205d9"
I20260620 03:03:53.974584 19846 raft_consensus.cc:3037] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:53.974618 20113 raft_consensus.cc:3037] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:53.976737 20113 raft_consensus.cc:2454] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 611e720a917f4f0096f7a646622bd880 in term 1.
I20260620 03:03:53.976747 19846 raft_consensus.cc:2454] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 611e720a917f4f0096f7a646622bd880 in term 1.
I20260620 03:03:53.977165 19914 leader_election.cc:302] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 611e720a917f4f0096f7a646622bd880, 6f2b5808bd8d45ff847a4971f66205d9; no voters:
I20260620 03:03:53.977519 20183 raft_consensus.cc:2781] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:03:53.978000 20183 raft_consensus.cc:686] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [term 1 LEADER]: Becoming Leader. State: Replica: 611e720a917f4f0096f7a646622bd880, State: Running, Role: LEADER
I20260620 03:03:53.978479 20183 consensus_queue.cc:237] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "611e720a917f4f0096f7a646622bd880" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44261 } } peers { permanent_uuid: "6f2b5808bd8d45ff847a4971f66205d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38037 } } peers { permanent_uuid: "09ed1345bb1949469dbb614f0f4e59c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45373 } }
I20260620 03:03:53.982897 19701 catalog_manager.cc:5495] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 reported cstate change: term changed from 0 to 1, leader changed from <none> to 611e720a917f4f0096f7a646622bd880 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "611e720a917f4f0096f7a646622bd880" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "611e720a917f4f0096f7a646622bd880" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f2b5808bd8d45ff847a4971f66205d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09ed1345bb1949469dbb614f0f4e59c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45373 } health_report { overall_health: UNKNOWN } } }
I20260620 03:03:54.020241 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:54.015082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60326 (local address 127.0.97.2:44261)
0620 03:03:54.015304 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:54.015309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:54.015327 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:03:54.015490 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:54.015494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:54.015575 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:54.015694 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:03:54.016118 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:54.016785 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:54.018273 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:54.018679 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:54.018776 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:54.019178 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:03:54.019679 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:03:54.019785 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:54.019864 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 03:03:54.019927 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 03:03:54.028120 20113 raft_consensus.cc:1261] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9 [term 1 FOLLOWER]: Refusing update from remote peer 611e720a917f4f0096f7a646622bd880: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:03:54.028133 19846 raft_consensus.cc:1261] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5 [term 1 FOLLOWER]: Refusing update from remote peer 611e720a917f4f0096f7a646622bd880: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:03:54.029078 20187 consensus_queue.cc:1035] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f2b5808bd8d45ff847a4971f66205d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:03:54.029417 20183 consensus_queue.cc:1035] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ed1345bb1949469dbb614f0f4e59c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:03:54.041721 20190 mvcc.cc:205] Tried to move back new op lower bound from 7298763300969709568 to 7298763300779700224. Current Snapshot: MvccSnapshot[applied={T|T < 7298763300969709568}]
I20260620 03:03:54.047734 20192 mvcc.cc:205] Tried to move back new op lower bound from 7298763300969709568 to 7298763300779700224. Current Snapshot: MvccSnapshot[applied={T|T < 7298763300969709568}]
I20260620 03:03:54.055882 19702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:03:54.056205 19702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:03:54.059576 19702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc0b86edce594b43bc7169231360b03b: Sending DeleteTablet for 3 replicas of tablet 131cb55aeb46444d9fd49d41bad976bc
I20260620 03:03:54.060593 19960 tablet_service.cc:1514] Processing DeleteTablet for tablet 131cb55aeb46444d9fd49d41bad976bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:03:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60322
I20260620 03:03:54.060779 20093 tablet_service.cc:1514] Processing DeleteTablet for tablet 131cb55aeb46444d9fd49d41bad976bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:03:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32882
I20260620 03:03:54.060982 19826 tablet_service.cc:1514] Processing DeleteTablet for tablet 131cb55aeb46444d9fd49d41bad976bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:03:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47284
I20260620 03:03:54.062817 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 19763
I20260620 03:03:54.071976 20202 tablet_replica.cc:330] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5: stopping tablet replica
I20260620 03:03:54.072257 20200 tablet_replica.cc:330] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880: stopping tablet replica
I20260620 03:03:54.072789 20200 raft_consensus.cc:2229] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:03:54.073280 20202 raft_consensus.cc:2229] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:03:54.073494 20200 raft_consensus.cc:2258] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:03:54.074258 20202 raft_consensus.cc:2258] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:03:54.075301 20201 tablet_replica.cc:330] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9: stopping tablet replica
I20260620 03:03:54.075973 20200 ts_tablet_manager.cc:1905] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:03:54.077652 20202 ts_tablet_manager.cc:1905] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:03:54.078111 20201 raft_consensus.cc:2229] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:03:54.078665 20201 raft_consensus.cc:2258] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:03:54.083823 20200 ts_tablet_manager.cc:1918] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:03:54.084040 20200 log.cc:1196] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/131cb55aeb46444d9fd49d41bad976bc
I20260620 03:03:54.084604 20200 ts_tablet_manager.cc:1939] T 131cb55aeb46444d9fd49d41bad976bc P 611e720a917f4f0096f7a646622bd880: Deleting consensus metadata
I20260620 03:03:54.086304 20202 ts_tablet_manager.cc:1918] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:03:54.086536 20202 log.cc:1196] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-0/wal/wals/131cb55aeb46444d9fd49d41bad976bc
I20260620 03:03:54.086467 19687 catalog_manager.cc:4841] TS 611e720a917f4f0096f7a646622bd880 (127.0.97.2:44261): tablet 131cb55aeb46444d9fd49d41bad976bc (table test-table [id=3e950ff94d0040b3ab657a122a35b73d]) successfully deleted
I20260620 03:03:54.087131 20202 ts_tablet_manager.cc:1939] T 131cb55aeb46444d9fd49d41bad976bc P 09ed1345bb1949469dbb614f0f4e59c5: Deleting consensus metadata
I20260620 03:03:54.087620 20201 ts_tablet_manager.cc:1905] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:03:54.088963 19686 catalog_manager.cc:4841] TS 09ed1345bb1949469dbb614f0f4e59c5 (127.0.97.1:45373): tablet 131cb55aeb46444d9fd49d41bad976bc (table test-table [id=3e950ff94d0040b3ab657a122a35b73d]) successfully deleted
I20260620 03:03:54.094741 20201 ts_tablet_manager.cc:1918] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:03:54.094925 20201 log.cc:1196] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/131cb55aeb46444d9fd49d41bad976bc
I20260620 03:03:54.095496 20201 ts_tablet_manager.cc:1939] T 131cb55aeb46444d9fd49d41bad976bc P 6f2b5808bd8d45ff847a4971f66205d9: Deleting consensus metadata
I20260620 03:03:54.097476 19687 catalog_manager.cc:4841] TS 6f2b5808bd8d45ff847a4971f66205d9 (127.0.97.3:38037): tablet 131cb55aeb46444d9fd49d41bad976bc (table test-table [id=3e950ff94d0040b3ab657a122a35b73d]) successfully deleted
Jun 20 03:03:54 dist-test-slave-f1j0 krb5kdc[19657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 03:03:54.109433 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:54.064122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47294 (local address 127.0.97.1:45373)
0620 03:03:54.064301 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:54.064307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:54.087807 (+ 23500us) server_negotiation.cc:407] Connection header received
0620 03:03:54.089526 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:54.089532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:54.089637 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:54.089708 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:54.091479 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:54.092585 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:54.098751 (+ 6166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:54.099194 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:54.099356 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:54.103544 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:54.103562 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:54.103577 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:54.103619 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:54.106603 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:54.107318 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:54.107326 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:54.107329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:54.107466 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:54.107862 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:54.107867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:54.107869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:54.108217 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:54.108539 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:54.108838 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 03:03:54.109062 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:03:54.110193 19886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:54.210301 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 19763
I20260620 03:03:54.234310 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 19897
Jun 20 03:03:54 dist-test-slave-f1j0 krb5kdc[19657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 03:03:54.253153 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:54.235100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60340 (local address 127.0.97.2:44261)
0620 03:03:54.235288 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:54.235294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:54.235474 (+ 180us) server_negotiation.cc:407] Connection header received
0620 03:03:54.235673 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:54.235678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:54.235755 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:54.235854 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:54.237537 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:54.238804 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:54.240671 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:54.241357 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:54.241491 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:54.246778 (+ 5287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:54.246792 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:54.246806 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:54.246844 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:54.250391 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:54.251117 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:54.251124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:54.251127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:54.251326 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:54.251754 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:54.251758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:54.251759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:54.252091 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:54.252332 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:54.252684 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 03:03:54.252836 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 03:03:54.254015 20020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:54.369313 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 19897
I20260620 03:03:54.390699 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 20030
Jun 20 03:03:54 dist-test-slave-f1j0 krb5kdc[19657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 03:03:54.405121 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:54.391478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.0.97.3:38037)
0620 03:03:54.391690 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:54.391695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:54.391713 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:03:54.391852 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:54.391856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:54.391922 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:54.392007 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:54.393258 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:54.394104 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:54.395375 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:54.395777 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:54.395872 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:54.400154 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:54.400167 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:54.400183 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:54.400219 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:54.402896 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:54.403519 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:54.403535 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:54.403537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:54.403671 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:54.403967 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:54.403970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:54.403972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:54.404288 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:54.404505 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:54.404727 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 03:03:54.404875 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 03:03:54.405990 20153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:54.494596 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 20030
I20260620 03:03:54.511401 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 19670
I20260620 03:03:54.512511 19732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:54.623024 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 19670
2026-06-20T03:03:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4333 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T03:03:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:03:54Z Disabled control of system clock
I20260620 03:03:54.701742 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40481
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 03:03:55.035712 20211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:55.035991 20211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:55.036073 20211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:55.043270 20211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:03:55.043380 20211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:55.043481 20211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:55.043581 20211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:03:55.043663 20211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:03:55.052903 20211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40481
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:55.054342 20211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:55.056176 20211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:55.067902 20216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:55.067876 20217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:55.067950 20219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:55.068670 20211 server_base.cc:1034] running on GCE node
I20260620 03:03:55.069463 20211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:55.070880 20211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:55.072111 20211 hybrid_clock.cc:648] HybridClock initialized: now 1781924635072068 us; error 37 us; skew 500 ppm
I20260620 03:03:55.072450 20211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:03:55.074764 20211 webserver.cc:460] Webserver started at http://127.0.97.62:40031/ using document root <none> and password file <none>
I20260620 03:03:55.075464 20211 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:55.075600 20211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:55.075923 20211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:55.078935 20211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "f4389d6de801403ea9b497448b0a5ff9"
format_stamp: "Formatted at 2026-06-20 03:03:55 on dist-test-slave-f1j0"
server_key: "2bf20535a5588536c96c475302266352"
server_key_iv: "d0d78cea87cd7091ef6d454d0bffc207"
server_key_version: "encryptionkey@0"
I20260620 03:03:55.079738 20211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "f4389d6de801403ea9b497448b0a5ff9"
format_stamp: "Formatted at 2026-06-20 03:03:55 on dist-test-slave-f1j0"
server_key: "2bf20535a5588536c96c475302266352"
server_key_iv: "d0d78cea87cd7091ef6d454d0bffc207"
server_key_version: "encryptionkey@0"
I20260620 03:03:55.084657 20211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 03:03:55.087952 20225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:55.089210 20211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:03:55.089403 20211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "f4389d6de801403ea9b497448b0a5ff9"
format_stamp: "Formatted at 2026-06-20 03:03:55 on dist-test-slave-f1j0"
server_key: "2bf20535a5588536c96c475302266352"
server_key_iv: "d0d78cea87cd7091ef6d454d0bffc207"
server_key_version: "encryptionkey@0"
I20260620 03:03:55.089660 20211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:55.112885 20211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:55.113822 20211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:55.114114 20211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:55.132711 20211 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40481
I20260620 03:03:55.132737 20276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40481 every 8 connection(s)
I20260620 03:03:55.134152 20211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:03:55.136313 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 20211
I20260620 03:03:55.136488 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:03:55.137004 388 external_mini_cluster.cc:1428] Setting key 01d82f1f8f72af1ce3466d79280c4978
I20260620 03:03:55.138818 20277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:55.150424 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:55.139000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54510 (local address 127.0.97.62:40481)
0620 03:03:55.139993 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:55.140004 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:55.140043 (+ 39us) server_negotiation.cc:407] Connection header received
0620 03:03:55.141744 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:55.141768 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:55.142245 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:55.142665 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:55.143225 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:55.144580 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:55.146005 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:55.146685 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:55.146881 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:55.148096 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:55.148104 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:55.148121 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:55.148681 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:55.148755 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:55.148776 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:55.148901 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:55.149117 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 03:03:55.149438 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":332,"threads_started":1}
I20260620 03:03:55.150432 20277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9: Bootstrap starting.
I20260620 03:03:55.153820 20277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:55.154834 20277 log.cc:824] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:55.157385 20277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9: No bootstrap required, opened a new log
I20260620 03:03:55.164812 20277 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4389d6de801403ea9b497448b0a5ff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40481 } }
I20260620 03:03:55.165329 20277 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:55.165463 20277 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4389d6de801403ea9b497448b0a5ff9, State: Initialized, Role: FOLLOWER
I20260620 03:03:55.166148 20277 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4389d6de801403ea9b497448b0a5ff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40481 } }
I20260620 03:03:55.166393 20277 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:03:55.166555 20277 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:03:55.166735 20277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:55.168891 20277 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4389d6de801403ea9b497448b0a5ff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40481 } }
I20260620 03:03:55.169426 20277 leader_election.cc:302] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4389d6de801403ea9b497448b0a5ff9; no voters:
I20260620 03:03:55.170082 20277 leader_election.cc:288] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:03:55.170243 20282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:03:55.170951 20282 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9 [term 1 LEADER]: Becoming Leader. State: Replica: f4389d6de801403ea9b497448b0a5ff9, State: Running, Role: LEADER
I20260620 03:03:55.171530 20282 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4389d6de801403ea9b497448b0a5ff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40481 } }
I20260620 03:03:55.172307 20277 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:03:55.174499 20283 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4389d6de801403ea9b497448b0a5ff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4389d6de801403ea9b497448b0a5ff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40481 } } }
I20260620 03:03:55.174865 20283 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9 [sys.catalog]: This master's current role is: LEADER
I20260620 03:03:55.175323 20284 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4389d6de801403ea9b497448b0a5ff9. Latest consensus state: current_term: 1 leader_uuid: "f4389d6de801403ea9b497448b0a5ff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4389d6de801403ea9b497448b0a5ff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40481 } } }
I20260620 03:03:55.175632 20284 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9 [sys.catalog]: This master's current role is: LEADER
I20260620 03:03:55.178833 20288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:03:55.184926 20288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:03:55.194336 20288 catalog_manager.cc:1347] Generated new cluster ID: cd6cd284eaef4cb1a26722e9bb1c5497
I20260620 03:03:55.194463 20288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:03:55.223503 20288 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:03:55.224825 20288 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:03:55.232324 20288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9: Generated new TSK 0
I20260620 03:03:55.233183 20288 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:03:55.238660 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40481
--builtin_ntp_servers=127.0.97.20:39823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 03:03:55.573210 20301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:55.573554 20301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:55.573735 20301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:55.581076 20301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:55.581168 20301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:55.581346 20301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 03:03:55.590646 20301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:55.591965 20301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:55.594038 20301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:55.607193 20307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:55.607216 20309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:55.607240 20306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:55.608067 20301 server_base.cc:1034] running on GCE node
I20260620 03:03:55.608606 20301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:55.609500 20301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:55.610760 20301 hybrid_clock.cc:648] HybridClock initialized: now 1781924635610707 us; error 46 us; skew 500 ppm
I20260620 03:03:55.611119 20301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:03:55.613329 20301 webserver.cc:460] Webserver started at http://127.0.97.1:38689/ using document root <none> and password file <none>
I20260620 03:03:55.614097 20301 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:55.614212 20301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:55.614568 20301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:55.617499 20301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-0/data/instance:
uuid: "7e7002f2cab0465e8d350fcad8f43336"
format_stamp: "Formatted at 2026-06-20 03:03:55 on dist-test-slave-f1j0"
server_key: "a0849fa739c2fd0b910f94a8c9a98e27"
server_key_iv: "845acdb6ed2ef08e6bda592afd78be3b"
server_key_version: "encryptionkey@0"
I20260620 03:03:55.618299 20301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7e7002f2cab0465e8d350fcad8f43336"
format_stamp: "Formatted at 2026-06-20 03:03:55 on dist-test-slave-f1j0"
server_key: "a0849fa739c2fd0b910f94a8c9a98e27"
server_key_iv: "845acdb6ed2ef08e6bda592afd78be3b"
server_key_version: "encryptionkey@0"
I20260620 03:03:55.623363 20301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:03:55.629814 20315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:55.631280 20301 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 03:03:55.631469 20301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-0/wal
uuid: "7e7002f2cab0465e8d350fcad8f43336"
format_stamp: "Formatted at 2026-06-20 03:03:55 on dist-test-slave-f1j0"
server_key: "a0849fa739c2fd0b910f94a8c9a98e27"
server_key_iv: "845acdb6ed2ef08e6bda592afd78be3b"
server_key_version: "encryptionkey@0"
I20260620 03:03:55.631700 20301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:55.647715 20301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:55.648581 20301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:55.648870 20301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:55.649963 20301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:55.651520 20301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:55.651639 20301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:55.651775 20301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:55.651860 20301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:55.684968 20301 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42831
I20260620 03:03:55.685024 20427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42831 every 8 connection(s)
I20260620 03:03:55.686426 20301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 03:03:55.696334 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 20301
I20260620 03:03:55.696532 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-0/wal/instance
I20260620 03:03:55.697080 388 external_mini_cluster.cc:1428] Setting key 8aaeb58d13e8d721bb25be82e383a40d
I20260620 03:03:55.700132 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:55.689208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60445 (local address 127.0.97.62:40481)
0620 03:03:55.689706 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:55.689713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:55.690927 (+ 1214us) server_negotiation.cc:407] Connection header received
0620 03:03:55.692850 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:55.692854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:55.692927 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:55.693030 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:55.694381 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:55.695165 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:55.696891 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:55.697531 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:55.697686 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:55.699237 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:55.699242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:55.699247 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:55.699270 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:55.699320 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:55.699336 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:55.699433 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:55.699653 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 03:03:55.699747 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":236,"threads_started":1}
I20260620 03:03:55.700300 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40481
--builtin_ntp_servers=127.0.97.20:39823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 03:03:55.701020 20431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:55.689895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40481 (local address 127.0.97.1:60445)
0620 03:03:55.690762 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:55.690803 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:55.692430 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:55.693408 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:55.693418 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:55.693519 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:03:55.694213 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:55.694233 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:55.695319 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:55.695323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:55.696751 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:55.696764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:55.697800 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:55.697808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:55.698102 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:55.698845 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:03:55.698850 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:03:55.698908 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:03:55.699016 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:03:55.699439 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:03:55.699448 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:03:55.699454 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 03:03:55.699604 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 03:03:55.699895 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":262,"threads_started":1}
I20260620 03:03:55.702481 20428 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40481
I20260620 03:03:55.702790 20428 heartbeater.cc:464] Registering TS with master...
I20260620 03:03:55.703668 20428 heartbeater.cc:511] Master 127.0.97.62:40481 requested a full tablet report, sending...
I20260620 03:03:55.705781 20242 ts_manager.cc:194] Registered new tserver with Master: 7e7002f2cab0465e8d350fcad8f43336 (127.0.97.1:42831)
I20260620 03:03:55.707242 20242 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:60445
W20260620 03:03:56.027530 20432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:56.027858 20432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:56.028009 20432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:56.035158 20432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:56.035291 20432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:56.035463 20432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 03:03:56.044348 20432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:56.045621 20432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:56.047390 20432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:56.060477 20437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:56.060384 20440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:56.060444 20438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:56.061319 20432 server_base.cc:1034] running on GCE node
I20260620 03:03:56.062027 20432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:56.063004 20432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:56.064240 20432 hybrid_clock.cc:648] HybridClock initialized: now 1781924636064182 us; error 51 us; skew 500 ppm
I20260620 03:03:56.064589 20432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:03:56.066890 20432 webserver.cc:460] Webserver started at http://127.0.97.2:42303/ using document root <none> and password file <none>
I20260620 03:03:56.067622 20432 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:56.067751 20432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:56.068078 20432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:56.071026 20432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-1/data/instance:
uuid: "fa5b6b9f273f4b838b8dbe32dbe77e3f"
format_stamp: "Formatted at 2026-06-20 03:03:56 on dist-test-slave-f1j0"
server_key: "8f56bcd6e0d267bade2e26875a756f3d"
server_key_iv: "fdee5411d883f2f94634084e587d8f8d"
server_key_version: "encryptionkey@0"
I20260620 03:03:56.071811 20432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fa5b6b9f273f4b838b8dbe32dbe77e3f"
format_stamp: "Formatted at 2026-06-20 03:03:56 on dist-test-slave-f1j0"
server_key: "8f56bcd6e0d267bade2e26875a756f3d"
server_key_iv: "fdee5411d883f2f94634084e587d8f8d"
server_key_version: "encryptionkey@0"
I20260620 03:03:56.077024 20432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 03:03:56.080626 20446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:56.082077 20432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:03:56.082253 20432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-1/wal
uuid: "fa5b6b9f273f4b838b8dbe32dbe77e3f"
format_stamp: "Formatted at 2026-06-20 03:03:56 on dist-test-slave-f1j0"
server_key: "8f56bcd6e0d267bade2e26875a756f3d"
server_key_iv: "fdee5411d883f2f94634084e587d8f8d"
server_key_version: "encryptionkey@0"
I20260620 03:03:56.082511 20432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:56.096625 20432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:56.097450 20432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:56.097771 20432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:56.098723 20432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:56.100190 20432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:56.100310 20432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:56.100445 20432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:56.100533 20432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:56.139093 20432 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34311
I20260620 03:03:56.139142 20558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34311 every 8 connection(s)
I20260620 03:03:56.140688 20432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 03:03:56.145902 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 20432
I20260620 03:03:56.146066 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-1/wal/instance
I20260620 03:03:56.146605 388 external_mini_cluster.cc:1428] Setting key a57c96fccaf84d90f4040cad705f4517
I20260620 03:03:56.149856 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40481
--builtin_ntp_servers=127.0.97.20:39823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 03:03:56.154204 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:56.143477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44819 (local address 127.0.97.62:40481)
0620 03:03:56.143661 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:56.143665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:56.145278 (+ 1613us) server_negotiation.cc:407] Connection header received
0620 03:03:56.147391 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:56.147397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:56.147477 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:56.147597 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:56.148982 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.149800 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.151420 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.151833 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.151961 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:56.153333 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:56.153337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:56.153341 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:56.153359 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:56.153403 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:56.153414 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:56.153516 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:56.153786 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 03:03:56.153884 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 03:03:56.155315 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:56.144167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40481 (local address 127.0.97.2:44819)
0620 03:03:56.145112 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:56.145153 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:56.147065 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:56.147868 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:56.147879 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:56.147997 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:03:56.148819 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:56.148840 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.149938 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.149945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:56.151262 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:56.151274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.152071 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.152082 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:56.152290 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:56.153004 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:03:56.153007 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:03:56.153046 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:03:56.153132 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:03:56.153516 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:03:56.153530 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:03:56.153540 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 03:03:56.153779 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 03:03:56.154190 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":311,"threads_started":1}
I20260620 03:03:56.156745 20559 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40481
I20260620 03:03:56.157065 20559 heartbeater.cc:464] Registering TS with master...
I20260620 03:03:56.157876 20559 heartbeater.cc:511] Master 127.0.97.62:40481 requested a full tablet report, sending...
I20260620 03:03:56.159255 20242 ts_manager.cc:194] Registered new tserver with Master: fa5b6b9f273f4b838b8dbe32dbe77e3f (127.0.97.2:34311)
I20260620 03:03:56.160073 20242 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:44819
W20260620 03:03:56.480971 20562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:56.481233 20562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:56.481393 20562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:56.488811 20562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:56.488910 20562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:56.489108 20562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 03:03:56.498636 20562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:56.499986 20562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:56.501835 20562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:56.515393 20567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:56.515470 20570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:56.515445 20568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:56.516001 20562 server_base.cc:1034] running on GCE node
I20260620 03:03:56.516587 20562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:56.517624 20562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:56.518829 20562 hybrid_clock.cc:648] HybridClock initialized: now 1781924636518770 us; error 57 us; skew 500 ppm
I20260620 03:03:56.519093 20562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 03:03:56.524435 20562 webserver.cc:460] Webserver started at http://127.0.97.3:42527/ using document root <none> and password file <none>
I20260620 03:03:56.525122 20562 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:56.525210 20562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:56.525475 20562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:56.528389 20562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-2/data/instance:
uuid: "ebda6c9cfb57400c8db3b5a34611651e"
format_stamp: "Formatted at 2026-06-20 03:03:56 on dist-test-slave-f1j0"
server_key: "f7ab700169fa764d5c24660dfd97612e"
server_key_iv: "d9663cf3b3bb9261ba8742894a4b37c4"
server_key_version: "encryptionkey@0"
I20260620 03:03:56.529100 20562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ebda6c9cfb57400c8db3b5a34611651e"
format_stamp: "Formatted at 2026-06-20 03:03:56 on dist-test-slave-f1j0"
server_key: "f7ab700169fa764d5c24660dfd97612e"
server_key_iv: "d9663cf3b3bb9261ba8742894a4b37c4"
server_key_version: "encryptionkey@0"
I20260620 03:03:56.533958 20562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 03:03:56.537407 20576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:56.538749 20562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:03:56.538866 20562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-2/wal
uuid: "ebda6c9cfb57400c8db3b5a34611651e"
format_stamp: "Formatted at 2026-06-20 03:03:56 on dist-test-slave-f1j0"
server_key: "f7ab700169fa764d5c24660dfd97612e"
server_key_iv: "d9663cf3b3bb9261ba8742894a4b37c4"
server_key_version: "encryptionkey@0"
I20260620 03:03:56.539036 20562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:56.557235 20562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:56.558283 20562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:56.558525 20562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:56.559484 20562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 03:03:56.561228 20562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 03:03:56.561316 20562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:56.561506 20562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 03:03:56.561645 20562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:56.594129 20562 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37991
I20260620 03:03:56.594189 20688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37991 every 8 connection(s)
I20260620 03:03:56.595464 20562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 03:03:56.598469 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 20562
I20260620 03:03:56.598651 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-2/wal/instance
I20260620 03:03:56.599092 388 external_mini_cluster.cc:1428] Setting key dd815a2b43d05c67760e4c27d7bd4b04
I20260620 03:03:56.607357 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:56.598094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52777 (local address 127.0.97.62:40481)
0620 03:03:56.598344 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:56.598350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:56.599968 (+ 1618us) server_negotiation.cc:407] Connection header received
0620 03:03:56.601774 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:56.601779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:56.601854 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:56.601931 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:56.602905 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.603618 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.605049 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.605417 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.605503 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:56.606737 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:56.606742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:56.606745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:56.606762 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:56.606791 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:56.606805 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:56.606873 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:56.607059 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 03:03:56.607132 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 03:03:56.608304 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:56.598796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40481 (local address 127.0.97.3:52777)
0620 03:03:56.599813 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:56.599863 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:56.601476 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:56.602120 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:56.602128 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:56.602209 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:03:56.602771 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:56.602786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.603743 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.603747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:56.604933 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:56.604944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.605650 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.605656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:56.605842 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:56.606446 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 03:03:56.606448 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:03:56.606484 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 03:03:56.606568 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:03:56.606885 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:03:56.606892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:03:56.606898 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 03:03:56.607029 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 03:03:56.607284 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":320,"threads_started":1}
I20260620 03:03:56.609601 20689 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40481
I20260620 03:03:56.609885 20689 heartbeater.cc:464] Registering TS with master...
I20260620 03:03:56.610550 20689 heartbeater.cc:511] Master 127.0.97.62:40481 requested a full tablet report, sending...
I20260620 03:03:56.611903 20242 ts_manager.cc:194] Registered new tserver with Master: ebda6c9cfb57400c8db3b5a34611651e (127.0.97.3:37991)
I20260620 03:03:56.612968 20242 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:52777
I20260620 03:03:56.614042 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 03:03:56.623652 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:56.617998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.0.97.62:40481)
0620 03:03:56.618269 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:56.618276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:56.618556 (+ 280us) server_negotiation.cc:407] Connection header received
0620 03:03:56.618696 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:56.618701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:56.618794 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:56.618883 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:56.619316 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.619986 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.621227 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.621672 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.621764 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:56.622804 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:56.622808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:56.622812 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:56.622828 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:56.622858 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:56.622869 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:56.622938 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:56.623104 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 03:03:56.623174 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 03:03:56.627190 20242 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 03:03:56.633946 20242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 03:03:56.664273 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:56.646432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37991 (local address 127.0.0.1:40054)
0620 03:03:56.649989 (+ 3557us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:56.650014 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:56.650260 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:56.651474 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:56.651486 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:56.651527 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:56.652035 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:56.652050 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.656025 (+ 3975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.656033 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:56.660870 (+ 4837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:56.660894 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.663231 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.663242 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:56.663479 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:56.663551 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 03:03:56.663711 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 03:03:56.663919 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2191,"thread_start_us":2274,"threads_started":1}
I20260620 03:03:56.664368 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:56.647108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40054 (local address 127.0.97.3:37991)
0620 03:03:56.648146 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:56.648155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:56.650121 (+ 1966us) server_negotiation.cc:407] Connection header received
0620 03:03:56.650487 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:56.650499 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:56.650851 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:56.651051 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:56.652294 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.655783 (+ 3489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.661089 (+ 5306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.662595 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.662734 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:56.663090 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:56.663782 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 03:03:56.663972 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":862,"thread_start_us":704,"threads_started":1}
I20260620 03:03:56.666384 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:56.646069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34311 (local address 127.0.0.1:58842)
0620 03:03:56.647329 (+ 1260us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:56.647372 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:56.647533 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:56.659412 (+ 11879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:56.659417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:56.659445 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:56.659892 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:56.659903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.661681 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.661686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:56.663639 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:56.663654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.665595 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.665602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:56.665767 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:56.665802 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 03:03:56.665907 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 03:03:56.666050 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"mutex_wait_us":65,"thread_start_us":885,"threads_started":1}
I20260620 03:03:56.666435 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:56.646395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58842 (local address 127.0.97.2:34311)
0620 03:03:56.658766 (+ 12371us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:56.658772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:56.658796 (+ 24us) server_negotiation.cc:407] Connection header received
0620 03:03:56.658924 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:56.658932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:56.659190 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:56.660091 (+ 901us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:56.660279 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.661502 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.663890 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.665370 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.665500 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:56.665763 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:56.665962 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 03:03:56.666126 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12238,"thread_start_us":12148,"threads_started":1}
I20260620 03:03:56.667488 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:56.648256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42831 (local address 127.0.0.1:44056)
0620 03:03:56.659139 (+ 10883us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:56.659164 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:56.659319 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:56.660306 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:56.660311 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:56.660337 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:56.660708 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:56.660717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.662613 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.662619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:56.664876 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:56.664887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.666739 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.666745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:56.666911 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:56.666942 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 03:03:56.667037 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 03:03:56.667170 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10715,"mutex_wait_us":64,"thread_start_us":10085,"threads_started":1}
I20260620 03:03:56.667696 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:56.648527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44056 (local address 127.0.97.1:42831)
0620 03:03:56.659836 (+ 11309us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:56.659844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:56.659872 (+ 28us) server_negotiation.cc:407] Connection header received
0620 03:03:56.659951 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:56.659956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:56.660125 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:56.660319 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:56.660881 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.662429 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.665089 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.666566 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.666690 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:56.666930 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:56.667106 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 03:03:56.667288 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11167,"thread_start_us":11068,"threads_started":1}
I20260620 03:03:56.669144 20624 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7f4f0451874c59ab6cf6d2b86717d9 (DEFAULT_TABLE table=test-table [id=7e66b661ea024bcd93adf01cbf47098c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:03:56.670141 20494 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7f4f0451874c59ab6cf6d2b86717d9 (DEFAULT_TABLE table=test-table [id=7e66b661ea024bcd93adf01cbf47098c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:03:56.670732 20624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7f4f0451874c59ab6cf6d2b86717d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:56.671430 20494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7f4f0451874c59ab6cf6d2b86717d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:56.672930 20363 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7f4f0451874c59ab6cf6d2b86717d9 (DEFAULT_TABLE table=test-table [id=7e66b661ea024bcd93adf01cbf47098c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 03:03:56.674253 20363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7f4f0451874c59ab6cf6d2b86717d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:56.682265 20707 tablet_bootstrap.cc:492] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e: Bootstrap starting.
I20260620 03:03:56.686184 20707 tablet_bootstrap.cc:654] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:56.687795 20707 log.cc:824] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:56.696853 20707 tablet_bootstrap.cc:492] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e: No bootstrap required, opened a new log
I20260620 03:03:56.697353 20707 ts_tablet_manager.cc:1397] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260620 03:03:56.700152 20709 tablet_bootstrap.cc:492] T 2a7f4f0451874c59ab6cf6d2b86717d9 P 7e7002f2cab0465e8d350fcad8f43336: Bootstrap starting.
I20260620 03:03:56.702445 20708 tablet_bootstrap.cc:492] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f: Bootstrap starting.
I20260620 03:03:56.704468 20709 tablet_bootstrap.cc:654] T 2a7f4f0451874c59ab6cf6d2b86717d9 P 7e7002f2cab0465e8d350fcad8f43336: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:56.705775 20709 log.cc:824] T 2a7f4f0451874c59ab6cf6d2b86717d9 P 7e7002f2cab0465e8d350fcad8f43336: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:56.706063 20708 tablet_bootstrap.cc:654] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:56.707139 20708 log.cc:824] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:56.707860 20707 raft_consensus.cc:348] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b6b9f273f4b838b8dbe32dbe77e3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34311 } } peers { permanent_uuid: "ebda6c9cfb57400c8db3b5a34611651e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37991 } } peers { permanent_uuid: "7e7002f2cab0465e8d350fcad8f43336" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } }
I20260620 03:03:56.708842 20707 raft_consensus.cc:374] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:56.708961 20707 raft_consensus.cc:729] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebda6c9cfb57400c8db3b5a34611651e, State: Initialized, Role: FOLLOWER
I20260620 03:03:56.709738 20707 consensus_queue.cc:260] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b6b9f273f4b838b8dbe32dbe77e3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34311 } } peers { permanent_uuid: "ebda6c9cfb57400c8db3b5a34611651e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37991 } } peers { permanent_uuid: "7e7002f2cab0465e8d350fcad8f43336" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } }
I20260620 03:03:56.710141 20428 heartbeater.cc:503] Master 127.0.97.62:40481 was elected leader, sending a full tablet report...
I20260620 03:03:56.710562 20709 tablet_bootstrap.cc:492] T 2a7f4f0451874c59ab6cf6d2b86717d9 P 7e7002f2cab0465e8d350fcad8f43336: No bootstrap required, opened a new log
I20260620 03:03:56.710836 20709 ts_tablet_manager.cc:1397] T 2a7f4f0451874c59ab6cf6d2b86717d9 P 7e7002f2cab0465e8d350fcad8f43336: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 03:03:56.710868 20689 heartbeater.cc:503] Master 127.0.97.62:40481 was elected leader, sending a full tablet report...
I20260620 03:03:56.713063 20707 ts_tablet_manager.cc:1428] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260620 03:03:56.713918 20708 tablet_bootstrap.cc:492] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f: No bootstrap required, opened a new log
I20260620 03:03:56.714200 20708 ts_tablet_manager.cc:1397] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260620 03:03:56.721724 20709 raft_consensus.cc:348] T 2a7f4f0451874c59ab6cf6d2b86717d9 P 7e7002f2cab0465e8d350fcad8f43336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b6b9f273f4b838b8dbe32dbe77e3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34311 } } peers { permanent_uuid: "ebda6c9cfb57400c8db3b5a34611651e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37991 } } peers { permanent_uuid: "7e7002f2cab0465e8d350fcad8f43336" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } }
I20260620 03:03:56.722175 20709 raft_consensus.cc:374] T 2a7f4f0451874c59ab6cf6d2b86717d9 P 7e7002f2cab0465e8d350fcad8f43336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:56.722277 20709 raft_consensus.cc:729] T 2a7f4f0451874c59ab6cf6d2b86717d9 P 7e7002f2cab0465e8d350fcad8f43336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7002f2cab0465e8d350fcad8f43336, State: Initialized, Role: FOLLOWER
I20260620 03:03:56.722898 20709 consensus_queue.cc:260] T 2a7f4f0451874c59ab6cf6d2b86717d9 P 7e7002f2cab0465e8d350fcad8f43336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b6b9f273f4b838b8dbe32dbe77e3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34311 } } peers { permanent_uuid: "ebda6c9cfb57400c8db3b5a34611651e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37991 } } peers { permanent_uuid: "7e7002f2cab0465e8d350fcad8f43336" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } }
I20260620 03:03:56.724864 20709 ts_tablet_manager.cc:1428] T 2a7f4f0451874c59ab6cf6d2b86717d9 P 7e7002f2cab0465e8d350fcad8f43336: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260620 03:03:56.724646 20708 raft_consensus.cc:348] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b6b9f273f4b838b8dbe32dbe77e3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34311 } } peers { permanent_uuid: "ebda6c9cfb57400c8db3b5a34611651e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37991 } } peers { permanent_uuid: "7e7002f2cab0465e8d350fcad8f43336" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } }
I20260620 03:03:56.725091 20708 raft_consensus.cc:374] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:56.725199 20708 raft_consensus.cc:729] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5b6b9f273f4b838b8dbe32dbe77e3f, State: Initialized, Role: FOLLOWER
I20260620 03:03:56.725828 20708 consensus_queue.cc:260] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b6b9f273f4b838b8dbe32dbe77e3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34311 } } peers { permanent_uuid: "ebda6c9cfb57400c8db3b5a34611651e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37991 } } peers { permanent_uuid: "7e7002f2cab0465e8d350fcad8f43336" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } }
I20260620 03:03:56.726903 20559 heartbeater.cc:503] Master 127.0.97.62:40481 was elected leader, sending a full tablet report...
I20260620 03:03:56.727603 20708 ts_tablet_manager.cc:1428] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 03:03:56.742182 20715 raft_consensus.cc:482] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 03:03:56.742537 20715 raft_consensus.cc:504] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b6b9f273f4b838b8dbe32dbe77e3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34311 } } peers { permanent_uuid: "ebda6c9cfb57400c8db3b5a34611651e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37991 } } peers { permanent_uuid: "7e7002f2cab0465e8d350fcad8f43336" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } }
I20260620 03:03:56.744566 20715 leader_election.cc:288] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebda6c9cfb57400c8db3b5a34611651e (127.0.97.3:37991), 7e7002f2cab0465e8d350fcad8f43336 (127.0.97.1:42831)
I20260620 03:03:56.753391 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:56.744956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34513 (local address 127.0.97.1:42831)
0620 03:03:56.745156 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:56.745165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:56.746066 (+ 901us) server_negotiation.cc:407] Connection header received
0620 03:03:56.746184 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:56.746191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:56.746276 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:56.746361 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:56.746991 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.748018 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.751000 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.752226 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.752365 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:56.752461 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:56.752999 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 03:03:56.753137 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 03:03:56.753391 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:56.744955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43733 (local address 127.0.97.3:37991)
0620 03:03:56.745360 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:56.745369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:56.745770 (+ 401us) server_negotiation.cc:407] Connection header received
0620 03:03:56.745995 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:56.746007 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:56.746113 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:56.746213 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 03:03:56.746803 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.747906 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.751012 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.752238 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.752365 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:56.752464 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:56.752999 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 03:03:56.753132 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260620 03:03:56.753624 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:56.744714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37991 (local address 127.0.97.2:43733)
0620 03:03:56.745631 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:56.745680 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:56.745849 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:56.746215 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:56.746218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:56.746233 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:56.746657 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:56.746669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.748036 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.748040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:56.750775 (+ 2735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:56.750789 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.752478 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.752484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:56.752753 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:56.752852 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 03:03:56.752948 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 03:03:56.753171 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":284,"thread_start_us":524,"threads_started":1}
I20260620 03:03:56.753646 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:56.744988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42831 (local address 127.0.97.2:34513)
0620 03:03:56.745960 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:56.745975 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 03:03:56.746080 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:03:56.746404 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:03:56.746409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:03:56.746427 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 03:03:56.746856 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:56.746864 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.748146 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.748157 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:56.750774 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:03:56.750789 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.752469 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.752476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:03:56.752750 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:56.752854 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 03:03:56.752948 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 03:03:56.753171 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":253,"thread_start_us":671,"threads_started":1}
I20260620 03:03:56.754849 20644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7f4f0451874c59ab6cf6d2b86717d9" candidate_uuid: "fa5b6b9f273f4b838b8dbe32dbe77e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebda6c9cfb57400c8db3b5a34611651e" is_pre_election: true
I20260620 03:03:56.754849 20383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7f4f0451874c59ab6cf6d2b86717d9" candidate_uuid: "fa5b6b9f273f4b838b8dbe32dbe77e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7002f2cab0465e8d350fcad8f43336" is_pre_election: true
I20260620 03:03:56.755465 20383 raft_consensus.cc:2454] T 2a7f4f0451874c59ab6cf6d2b86717d9 P 7e7002f2cab0465e8d350fcad8f43336 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa5b6b9f273f4b838b8dbe32dbe77e3f in term 0.
I20260620 03:03:56.755465 20644 raft_consensus.cc:2454] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa5b6b9f273f4b838b8dbe32dbe77e3f in term 0.
I20260620 03:03:56.756208 20447 leader_election.cc:302] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ebda6c9cfb57400c8db3b5a34611651e, fa5b6b9f273f4b838b8dbe32dbe77e3f; no voters:
I20260620 03:03:56.756633 20715 raft_consensus.cc:2781] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 03:03:56.756780 20715 raft_consensus.cc:482] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 03:03:56.756887 20715 raft_consensus.cc:3037] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:56.759135 20715 raft_consensus.cc:504] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b6b9f273f4b838b8dbe32dbe77e3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34311 } } peers { permanent_uuid: "ebda6c9cfb57400c8db3b5a34611651e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37991 } } peers { permanent_uuid: "7e7002f2cab0465e8d350fcad8f43336" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } }
I20260620 03:03:56.759739 20715 leader_election.cc:288] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [CANDIDATE]: Term 1 election: Requested vote from peers ebda6c9cfb57400c8db3b5a34611651e (127.0.97.3:37991), 7e7002f2cab0465e8d350fcad8f43336 (127.0.97.1:42831)
I20260620 03:03:56.760200 20644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7f4f0451874c59ab6cf6d2b86717d9" candidate_uuid: "fa5b6b9f273f4b838b8dbe32dbe77e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebda6c9cfb57400c8db3b5a34611651e"
I20260620 03:03:56.760280 20383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7f4f0451874c59ab6cf6d2b86717d9" candidate_uuid: "fa5b6b9f273f4b838b8dbe32dbe77e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7002f2cab0465e8d350fcad8f43336"
I20260620 03:03:56.760426 20644 raft_consensus.cc:3037] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:56.760466 20383 raft_consensus.cc:3037] T 2a7f4f0451874c59ab6cf6d2b86717d9 P 7e7002f2cab0465e8d350fcad8f43336 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:56.763729 20644 raft_consensus.cc:2454] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa5b6b9f273f4b838b8dbe32dbe77e3f in term 1.
I20260620 03:03:56.763729 20383 raft_consensus.cc:2454] T 2a7f4f0451874c59ab6cf6d2b86717d9 P 7e7002f2cab0465e8d350fcad8f43336 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa5b6b9f273f4b838b8dbe32dbe77e3f in term 1.
I20260620 03:03:56.764149 20450 leader_election.cc:302] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7002f2cab0465e8d350fcad8f43336, fa5b6b9f273f4b838b8dbe32dbe77e3f; no voters:
I20260620 03:03:56.764462 20715 raft_consensus.cc:2781] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:03:56.764858 20715 raft_consensus.cc:686] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [term 1 LEADER]: Becoming Leader. State: Replica: fa5b6b9f273f4b838b8dbe32dbe77e3f, State: Running, Role: LEADER
I20260620 03:03:56.765376 20715 consensus_queue.cc:237] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b6b9f273f4b838b8dbe32dbe77e3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34311 } } peers { permanent_uuid: "ebda6c9cfb57400c8db3b5a34611651e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37991 } } peers { permanent_uuid: "7e7002f2cab0465e8d350fcad8f43336" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } }
I20260620 03:03:56.770273 20241 catalog_manager.cc:5495] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f reported cstate change: term changed from 0 to 1, leader changed from <none> to fa5b6b9f273f4b838b8dbe32dbe77e3f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "fa5b6b9f273f4b838b8dbe32dbe77e3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b6b9f273f4b838b8dbe32dbe77e3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebda6c9cfb57400c8db3b5a34611651e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e7002f2cab0465e8d350fcad8f43336" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } health_report { overall_health: UNKNOWN } } }
I20260620 03:03:56.809700 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:56.804627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.0.97.2:34311)
0620 03:03:56.804860 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:56.804864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:56.804882 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:03:56.805054 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:56.805058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:56.805142 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:56.805261 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:03:56.805616 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.806289 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.807588 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.808004 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.808095 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:56.808570 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:03:56.809048 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:03:56.809154 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:56.809270 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 03:03:56.809327 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 03:03:56.817731 20644 raft_consensus.cc:1261] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e [term 1 FOLLOWER]: Refusing update from remote peer fa5b6b9f273f4b838b8dbe32dbe77e3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:03:56.817731 20383 raft_consensus.cc:1261] T 2a7f4f0451874c59ab6cf6d2b86717d9 P 7e7002f2cab0465e8d350fcad8f43336 [term 1 FOLLOWER]: Refusing update from remote peer fa5b6b9f273f4b838b8dbe32dbe77e3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:03:56.818763 20715 consensus_queue.cc:1035] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebda6c9cfb57400c8db3b5a34611651e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:03:56.819085 20718 consensus_queue.cc:1035] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7002f2cab0465e8d350fcad8f43336" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:03:56.831447 20723 mvcc.cc:205] Tried to move back new op lower bound from 7298763312394997760 to 7298763312195477504. Current Snapshot: MvccSnapshot[applied={T|T < 7298763312394997760}]
I20260620 03:03:56.845033 20242 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:03:56.845346 20242 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 03:03:56.849274 20242 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4389d6de801403ea9b497448b0a5ff9: Sending DeleteTablet for 3 replicas of tablet 2a7f4f0451874c59ab6cf6d2b86717d9
I20260620 03:03:56.850450 20494 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7f4f0451874c59ab6cf6d2b86717d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:03:56 UTC) from {username='slave'} at 127.0.0.1:58842
I20260620 03:03:56.850746 20624 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7f4f0451874c59ab6cf6d2b86717d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:03:56 UTC) from {username='slave'} at 127.0.0.1:40054
I20260620 03:03:56.851637 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 20301
I20260620 03:03:56.851922 20363 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7f4f0451874c59ab6cf6d2b86717d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 03:03:56 UTC) from {username='slave'} at 127.0.0.1:44056
I20260620 03:03:56.863462 20731 tablet_replica.cc:330] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f: stopping tablet replica
I20260620 03:03:56.864213 20733 tablet_replica.cc:330] T 2a7f4f0451874c59ab6cf6d2b86717d9 P 7e7002f2cab0465e8d350fcad8f43336: stopping tablet replica
I20260620 03:03:56.864885 20732 tablet_replica.cc:330] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e: stopping tablet replica
I20260620 03:03:56.865348 20732 raft_consensus.cc:2229] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:03:56.865634 20733 raft_consensus.cc:2229] T 2a7f4f0451874c59ab6cf6d2b86717d9 P 7e7002f2cab0465e8d350fcad8f43336 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 03:03:56.865947 20732 raft_consensus.cc:2258] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:03:56.866143 20733 raft_consensus.cc:2258] T 2a7f4f0451874c59ab6cf6d2b86717d9 P 7e7002f2cab0465e8d350fcad8f43336 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:03:56.866499 20731 raft_consensus.cc:2229] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [term 1 LEADER]: Raft consensus shutting down.
I20260620 03:03:56.867198 20731 raft_consensus.cc:2258] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 03:03:56.868422 20732 ts_tablet_manager.cc:1905] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:03:56.868988 20731 ts_tablet_manager.cc:1905] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:03:56.874886 20733 ts_tablet_manager.cc:1905] T 2a7f4f0451874c59ab6cf6d2b86717d9 P 7e7002f2cab0465e8d350fcad8f43336: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 03:03:56.875514 20732 ts_tablet_manager.cc:1918] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:03:56.875753 20732 log.cc:1196] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-2/wal/wals/2a7f4f0451874c59ab6cf6d2b86717d9
I20260620 03:03:56.876369 20732 ts_tablet_manager.cc:1939] T 2a7f4f0451874c59ab6cf6d2b86717d9 P ebda6c9cfb57400c8db3b5a34611651e: Deleting consensus metadata
I20260620 03:03:56.877615 20731 ts_tablet_manager.cc:1918] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 03:03:56.878201 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:56.852998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.0.97.1:42831)
0620 03:03:56.853179 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:56.853186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:56.863253 (+ 10067us) server_negotiation.cc:407] Connection header received
0620 03:03:56.863797 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:56.863804 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:56.863973 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:56.864058 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:56.864769 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.866027 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.874200 (+ 8173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.874852 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.874975 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:56.876478 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:56.876489 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:56.876506 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:56.877235 (+ 729us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:56.877296 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:56.877323 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:56.877410 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:56.877743 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 03:03:56.877858 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 03:03:56.878404 20226 catalog_manager.cc:4841] TS ebda6c9cfb57400c8db3b5a34611651e (127.0.97.3:37991): tablet 2a7f4f0451874c59ab6cf6d2b86717d9 (table test-table [id=7e66b661ea024bcd93adf01cbf47098c]) successfully deleted
I20260620 03:03:56.878971 20731 log.cc:1196] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f: Deleting WAL directory at /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781924320356866-388-0/minicluster-data/ts-1/wal/wals/2a7f4f0451874c59ab6cf6d2b86717d9
I20260620 03:03:56.879401 20423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:56.879565 20731 ts_tablet_manager.cc:1939] T 2a7f4f0451874c59ab6cf6d2b86717d9 P fa5b6b9f273f4b838b8dbe32dbe77e3f: Deleting consensus metadata
I20260620 03:03:56.881279 20229 catalog_manager.cc:4841] TS fa5b6b9f273f4b838b8dbe32dbe77e3f (127.0.97.2:34311): tablet 2a7f4f0451874c59ab6cf6d2b86717d9 (table test-table [id=7e66b661ea024bcd93adf01cbf47098c]) successfully deleted
I20260620 03:03:56.969542 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 20301
W20260620 03:03:56.985028 20229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42831 (error 108)
I20260620 03:03:56.985185 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 20432
I20260620 03:03:56.987176 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:56.986679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42831 (local address 127.0.0.1:44078)
0620 03:03:56.986870 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:56.987026 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 03:03:56.987547 20229 catalog_manager.cc:4584] TS 7e7002f2cab0465e8d350fcad8f43336 (127.0.97.1:42831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a7f4f0451874c59ab6cf6d2b86717d9: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42831: connect: Connection refused (error 111)
I20260620 03:03:56.993517 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:56.986343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58854 (local address 127.0.97.2:34311)
0620 03:03:56.986529 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:56.986535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:56.986562 (+ 27us) server_negotiation.cc:407] Connection header received
0620 03:03:56.986617 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:56.986620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:56.986705 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:56.986840 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:56.987354 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.988587 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.989938 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:56.990631 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:56.990768 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:56.991739 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:56.991751 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:56.991769 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:56.992599 (+ 830us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:56.992666 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:56.992696 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:56.992815 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:56.993019 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 03:03:56.993127 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 03:03:56.994233 20554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:57.041350 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:57.040873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42831 (local address 127.0.0.1:44080)
0620 03:03:57.041097 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:57.041186 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 03:03:57.042550 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:57.042137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42831 (local address 127.0.0.1:44086)
0620 03:03:57.042281 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:57.042354 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 03:03:57.103310 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:57.102776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42831 (local address 127.0.0.1:44090)
0620 03:03:57.103032 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:57.103124 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 03:03:57.104324 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:57.103992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42831 (local address 127.0.0.1:44094)
0620 03:03:57.104143 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:57.104219 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 03:03:57.107074 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 20432
I20260620 03:03:57.127336 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 20562
I20260620 03:03:57.134191 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:57.128072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.0.97.3:37991)
0620 03:03:57.128236 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:57.128240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:57.128311 (+ 71us) server_negotiation.cc:407] Connection header received
0620 03:03:57.128454 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:57.128458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:57.128550 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:57.128634 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:57.128990 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:57.129767 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:57.131013 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:57.131391 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:57.131480 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:57.132611 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:57.132624 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:57.132649 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:03:57.133385 (+ 736us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:57.133442 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:03:57.133469 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:57.133718 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:57.133763 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 03:03:57.133865 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 03:03:57.134912 20684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:57.185465 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:57.184988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42831 (local address 127.0.0.1:44106)
0620 03:03:57.185214 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:57.185308 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 03:03:57.186607 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:57.186236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42831 (local address 127.0.0.1:44114)
0620 03:03:57.186417 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 03:03:57.186484 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 03:03:57.222719 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 20562
I20260620 03:03:57.239250 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 20211
I20260620 03:03:57.240423 20272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:57.349871 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 20211
2026-06-20T03:03:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2725 ms)
[----------] 6 tests from EncryptionPolicyTest (21599 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:03:57 dist-test-slave-f1j0 krb5kdc[20742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:03:57 dist-test-slave-f1j0 krb5kdc[20742](info): set up 2 sockets
Jun 20 03:03:57 dist-test-slave-f1j0 krb5kdc[20742](info): commencing operation
krb5kdc: starting...
W20260620 03:03:58.462080 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:03:58 dist-test-slave-f1j0 krb5kdc[20742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:03:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:03:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:03:58.721279 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:03:59.047341 20755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:03:59.047686 20755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:03:59.047845 20755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:03:59.055011 20755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:03:59.055171 20755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:03:59.055272 20755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:03:59.055377 20755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:03:59.055450 20755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:03:59.065088 20755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38351
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:03:59.066443 20755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:03:59.067736 20755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:03:59.079368 20763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:59.079396 20760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:03:59.079334 20761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:03:59.080534 20755 server_base.cc:1034] running on GCE node
I20260620 03:03:59.081276 20755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:03:59.082643 20755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:03:59.083832 20755 hybrid_clock.cc:648] HybridClock initialized: now 1781924639083800 us; error 29 us; skew 500 ppm
I20260620 03:03:59.084125 20755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:03:59 dist-test-slave-f1j0 krb5kdc[20742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:03:59.087059 20755 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:03:59.088562 20755 webserver.cc:460] Webserver started at http://127.0.0.1:33115/ using document root <none> and password file <none>
I20260620 03:03:59.089219 20755 fs_manager.cc:359] Metadata directory not provided
I20260620 03:03:59.089359 20755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:03:59.089742 20755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:03:59.092639 20755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "be78a4b749ba44b5a678b00561660199"
format_stamp: "Formatted at 2026-06-20 03:03:59 on dist-test-slave-f1j0"
server_key: "92182c6fd1d1cf2e8cb6b6864f782adf"
server_key_iv: "1f9c01f3def3a5303b60b559475a2a0e"
server_key_version: "encryptionkey@0"
I20260620 03:03:59.093390 20755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "be78a4b749ba44b5a678b00561660199"
format_stamp: "Formatted at 2026-06-20 03:03:59 on dist-test-slave-f1j0"
server_key: "92182c6fd1d1cf2e8cb6b6864f782adf"
server_key_iv: "1f9c01f3def3a5303b60b559475a2a0e"
server_key_version: "encryptionkey@0"
I20260620 03:03:59.098588 20755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:03:59.102064 20770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:03:59.103334 20755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 03:03:59.103513 20755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "be78a4b749ba44b5a678b00561660199"
format_stamp: "Formatted at 2026-06-20 03:03:59 on dist-test-slave-f1j0"
server_key: "92182c6fd1d1cf2e8cb6b6864f782adf"
server_key_iv: "1f9c01f3def3a5303b60b559475a2a0e"
server_key_version: "encryptionkey@0"
I20260620 03:03:59.103721 20755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:03:59.119400 20755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:03:59.123489 20755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:03:59.123876 20755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:03:59.143656 20755 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38351
I20260620 03:03:59.143685 20821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38351 every 8 connection(s)
I20260620 03:03:59.145064 20755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:03:59.148747 20822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:03:59.155040 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 20755
I20260620 03:03:59.155279 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:03:59.156111 388 external_mini_cluster.cc:1428] Setting key b8320645fbfbe504a69c9cac655200f5
I20260620 03:03:59.159971 20822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199: Bootstrap starting.
I20260620 03:03:59.163772 20822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199: Neither blocks nor log segments found. Creating new log.
I20260620 03:03:59.164837 20822 log.cc:824] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199: Log is configured to *not* fsync() on all Append() calls
I20260620 03:03:59.167945 20822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199: No bootstrap required, opened a new log
Jun 20 03:03:59 dist-test-slave-f1j0 krb5kdc[20742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 03:03:59.176954 20822 raft_consensus.cc:348] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be78a4b749ba44b5a678b00561660199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38351 } }
I20260620 03:03:59.177431 20822 raft_consensus.cc:374] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:03:59.177592 20822 raft_consensus.cc:729] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be78a4b749ba44b5a678b00561660199, State: Initialized, Role: FOLLOWER
I20260620 03:03:59.178246 20822 consensus_queue.cc:260] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be78a4b749ba44b5a678b00561660199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38351 } }
I20260620 03:03:59.178503 20822 raft_consensus.cc:388] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:03:59.178694 20822 raft_consensus.cc:482] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:03:59.178887 20822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:03:59.180768 20825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:59.158170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58869 (local address 127.0.0.1:38351)
0620 03:03:59.158953 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:59.158965 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:59.159006 (+ 41us) server_negotiation.cc:407] Connection header received
0620 03:03:59.160857 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:59.160881 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:59.161292 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:59.161729 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:59.163487 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:59.165132 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:59.166860 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:59.167602 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:59.167755 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:59.172526 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:59.172537 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:59.172557 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:59.172595 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:59.175821 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:59.176668 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:59.176675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:59.176681 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:59.176838 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:59.177311 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:59.177315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:59.177317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:59.178038 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:59.178375 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:59.178999 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 03:03:59.179341 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":373,"threads_started":1}
I20260620 03:03:59.181913 20822 raft_consensus.cc:504] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be78a4b749ba44b5a678b00561660199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38351 } }
I20260620 03:03:59.182439 20822 leader_election.cc:302] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be78a4b749ba44b5a678b00561660199; no voters:
I20260620 03:03:59.183080 20822 leader_election.cc:288] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:03:59.183198 20827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:03:59.183894 20827 raft_consensus.cc:686] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199 [term 1 LEADER]: Becoming Leader. State: Replica: be78a4b749ba44b5a678b00561660199, State: Running, Role: LEADER
I20260620 03:03:59.184502 20827 consensus_queue.cc:237] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be78a4b749ba44b5a678b00561660199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38351 } }
I20260620 03:03:59.185082 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:03:59.186496 20822 sys_catalog.cc:564] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:03:59.186514 20828 sys_catalog.cc:455] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be78a4b749ba44b5a678b00561660199. Latest consensus state: current_term: 1 leader_uuid: "be78a4b749ba44b5a678b00561660199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be78a4b749ba44b5a678b00561660199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38351 } } }
I20260620 03:03:59.186995 20828 sys_catalog.cc:458] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199 [sys.catalog]: This master's current role is: LEADER
I20260620 03:03:59.193380 20829 sys_catalog.cc:455] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be78a4b749ba44b5a678b00561660199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be78a4b749ba44b5a678b00561660199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38351 } } }
I20260620 03:03:59.195115 20829 sys_catalog.cc:458] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199 [sys.catalog]: This master's current role is: LEADER
I20260620 03:03:59.198742 20831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:03:59.207551 20831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:03:59.219772 20825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:03:59.200575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52007 (local address 127.0.0.1:38351)
0620 03:03:59.200746 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 03:03:59.200755 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:03:59.201129 (+ 374us) server_negotiation.cc:407] Connection header received
0620 03:03:59.201413 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:03:59.201420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:03:59.201524 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:03:59.201718 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:03:59.203836 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:59.205388 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:59.207256 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:03:59.207972 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:03:59.208123 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:03:59.211651 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:03:59.211659 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:03:59.211665 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:03:59.211703 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:03:59.215607 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:59.216599 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:59.216604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:59.216607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:59.216744 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:03:59.217333 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:03:59.217338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:03:59.217342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:03:59.217764 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:03:59.218015 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 03:03:59.219221 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0620 03:03:59.219410 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:03:59.228312 20831 catalog_manager.cc:1347] Generated new cluster ID: 2c6e49771d564de7b3df724ff3c4ba69
I20260620 03:03:59.228446 20831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:03:59.238903 20831 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:03:59.240356 20831 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:03:59.254251 20831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be78a4b749ba44b5a678b00561660199: Generated new TSK 0
I20260620 03:03:59.255095 20831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:03:59.263132 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 20755
I20260620 03:03:59.264359 20817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:03:59.356302 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 20755
W20260620 03:03:59.373791 20740 connection.cc:535] client connection to 127.0.0.1:38351 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38351): Connection reset by peer (error 104)
2026-06-20T03:03:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2008 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:03:59 dist-test-slave-f1j0 krb5kdc[20854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:03:59 dist-test-slave-f1j0 krb5kdc[20854](info): set up 2 sockets
Jun 20 03:03:59 dist-test-slave-f1j0 krb5kdc[20854](info): commencing operation
krb5kdc: starting...
W20260620 03:04:00.498427 388 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:04:00 dist-test-slave-f1j0 krb5kdc[20854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:04:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:04:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:04:00.756297 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46261
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:04:01.088245 20867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:01.088626 20867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:01.088800 20867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:01.096366 20867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:01.096542 20867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:01.096665 20867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:01.096766 20867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:01.096853 20867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:01.107023 20867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46261
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:01.108385 20867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:01.109814 20867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:01.121896 20872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:01.121846 20875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:01.121862 20873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:01.122853 20867 server_base.cc:1034] running on GCE node
I20260620 03:04:01.123656 20867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:01.125072 20867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:01.126309 20867 hybrid_clock.cc:648] HybridClock initialized: now 1781924641126237 us; error 59 us; skew 500 ppm
I20260620 03:04:01.126698 20867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:01 dist-test-slave-f1j0 krb5kdc[20854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:01.130115 20867 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:04:01.131659 20867 webserver.cc:460] Webserver started at http://127.0.0.1:43627/ using document root <none> and password file <none>
I20260620 03:04:01.132637 20867 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:01.132787 20867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:01.133147 20867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:01.137023 20867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "683a4f8ca84f410eb7798e5b68ba1e87"
format_stamp: "Formatted at 2026-06-20 03:04:01 on dist-test-slave-f1j0"
server_key: "a4045c6b7b36a66e30cec44ae47d2f6b"
server_key_iv: "6c25d6a75abdebe13ee39df1ce8e67cd"
server_key_version: "encryptionkey@0"
I20260620 03:04:01.137895 20867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "683a4f8ca84f410eb7798e5b68ba1e87"
format_stamp: "Formatted at 2026-06-20 03:04:01 on dist-test-slave-f1j0"
server_key: "a4045c6b7b36a66e30cec44ae47d2f6b"
server_key_iv: "6c25d6a75abdebe13ee39df1ce8e67cd"
server_key_version: "encryptionkey@0"
I20260620 03:04:01.143304 20867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 03:04:01.146966 20882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:01.148465 20867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:04:01.148675 20867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "683a4f8ca84f410eb7798e5b68ba1e87"
format_stamp: "Formatted at 2026-06-20 03:04:01 on dist-test-slave-f1j0"
server_key: "a4045c6b7b36a66e30cec44ae47d2f6b"
server_key_iv: "6c25d6a75abdebe13ee39df1ce8e67cd"
server_key_version: "encryptionkey@0"
I20260620 03:04:01.148892 20867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:01.185031 20867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:01.189754 20867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:01.190090 20867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:01.208901 20867 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46261
I20260620 03:04:01.208904 20933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46261 every 8 connection(s)
I20260620 03:04:01.210434 20867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:04:01.210754 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 20867
I20260620 03:04:01.210971 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:04:01.211654 388 external_mini_cluster.cc:1428] Setting key 8e2e7641511c8c441ae4ee60ce570541
I20260620 03:04:01.215113 20934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:01.224720 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:01.213520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53701 (local address 127.0.0.1:46261)
0620 03:04:01.214375 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:01.214388 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:01.214432 (+ 44us) server_negotiation.cc:407] Connection header received
0620 03:04:01.216187 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:01.216208 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:01.216598 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:01.216997 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:01.217546 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:01.218941 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:01.220253 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:01.220932 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:01.221125 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:01.222443 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:01.222450 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:01.222464 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:01.223052 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:01.223132 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:01.223159 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:01.223338 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:01.223610 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 03:04:01.223780 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":413,"threads_started":1}
I20260620 03:04:01.225977 20934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87: Bootstrap starting.
I20260620 03:04:01.228616 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:04:01.229215 20934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:01.230257 20934 log.cc:824] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:01.233866 20934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87: No bootstrap required, opened a new log
I20260620 03:04:01.245007 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:01.235261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36895 (local address 127.0.0.1:46261)
0620 03:04:01.235528 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:01.235535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:01.235578 (+ 43us) server_negotiation.cc:407] Connection header received
0620 03:04:01.235686 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:01.235693 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:01.235842 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:01.235995 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:01.237332 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:01.238880 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:01.240936 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:01.242165 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:01.242319 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:01.244014 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:01.244022 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:01.244032 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:01.244059 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:01.244136 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:01.244162 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:01.244309 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:01.244539 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 03:04:01.244601 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 03:04:01.247851 20934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683a4f8ca84f410eb7798e5b68ba1e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46261 } }
I20260620 03:04:01.248369 20934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:04:01.248484 20934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 683a4f8ca84f410eb7798e5b68ba1e87, State: Initialized, Role: FOLLOWER
I20260620 03:04:01.249329 20934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683a4f8ca84f410eb7798e5b68ba1e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46261 } }
I20260620 03:04:01.249730 20934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:04:01.249902 20934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:04:01.250110 20934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:01.253422 20934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683a4f8ca84f410eb7798e5b68ba1e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46261 } }
I20260620 03:04:01.254161 20934 leader_election.cc:302] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 683a4f8ca84f410eb7798e5b68ba1e87; no voters:
I20260620 03:04:01.254977 20934 leader_election.cc:288] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:04:01.255185 20944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:04:01.255949 20944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87 [term 1 LEADER]: Becoming Leader. State: Replica: 683a4f8ca84f410eb7798e5b68ba1e87, State: Running, Role: LEADER
I20260620 03:04:01.256562 20944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683a4f8ca84f410eb7798e5b68ba1e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46261 } }
I20260620 03:04:01.257427 20934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:01.259763 20945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "683a4f8ca84f410eb7798e5b68ba1e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683a4f8ca84f410eb7798e5b68ba1e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46261 } } }
I20260620 03:04:01.259785 20946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 683a4f8ca84f410eb7798e5b68ba1e87. Latest consensus state: current_term: 1 leader_uuid: "683a4f8ca84f410eb7798e5b68ba1e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683a4f8ca84f410eb7798e5b68ba1e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46261 } } }
I20260620 03:04:01.260226 20945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:01.260226 20946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:01.261097 20949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:04:01.267540 20949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:04:01.278285 20949 catalog_manager.cc:1347] Generated new cluster ID: a995b0d68b72492382cb331f3752aba5
I20260620 03:04:01.278560 20949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:04:01.307477 20949 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:04:01.309722 20949 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:04:01.322186 20949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 683a4f8ca84f410eb7798e5b68ba1e87: Generated new TSK 0
I20260620 03:04:01.323087 20949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:04:01.329391 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 20867
W20260620 03:04:01.330567 20929 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53701
W20260620 03:04:01.331297 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 20867: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:04:01.331502 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 20867
W20260620 03:04:01.349104 20852 connection.cc:535] client connection to 127.0.0.1:46261 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46261): Connection reset by peer (error 104)
2026-06-20T03:04:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1968 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:04:01 dist-test-slave-f1j0 krb5kdc[20965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:04:01 dist-test-slave-f1j0 krb5kdc[20965](info): set up 2 sockets
Jun 20 03:04:01 dist-test-slave-f1j0 krb5kdc[20965](info): commencing operation
krb5kdc: starting...
W20260620 03:04:02.449882 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:04:02 dist-test-slave-f1j0 krb5kdc[20965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:04:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:04:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:04:02.696785 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38971
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:04:03.021515 20978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:03.021834 20978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:03.021934 20978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:03.029125 20978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:03.029232 20978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:03.029278 20978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:03.029338 20978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:03.029383 20978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:03.039178 20978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38971
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:03.040356 20978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:03.041615 20978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:03.053355 20984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:03.053364 20986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:03.053411 20983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:03.054330 20978 server_base.cc:1034] running on GCE node
I20260620 03:04:03.055128 20978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:03.056535 20978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:03.057763 20978 hybrid_clock.cc:648] HybridClock initialized: now 1781924643057727 us; error 49 us; skew 500 ppm
I20260620 03:04:03.058108 20978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:03 dist-test-slave-f1j0 krb5kdc[20965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:03.064280 20978 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:04:03.065685 20978 webserver.cc:460] Webserver started at http://127.0.0.1:43963/ using document root <none> and password file <none>
I20260620 03:04:03.066323 20978 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:03.066416 20978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:03.066668 20978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:03.069543 20978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "192db2208a3c49c7b7355792f3a8cb27"
format_stamp: "Formatted at 2026-06-20 03:04:03 on dist-test-slave-f1j0"
server_key: "bf74f883b43b71672779a38e609fbfcb"
server_key_iv: "aaa7de43ef00d55988c210d4f7243481"
server_key_version: "encryptionkey@0"
I20260620 03:04:03.070303 20978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "192db2208a3c49c7b7355792f3a8cb27"
format_stamp: "Formatted at 2026-06-20 03:04:03 on dist-test-slave-f1j0"
server_key: "bf74f883b43b71672779a38e609fbfcb"
server_key_iv: "aaa7de43ef00d55988c210d4f7243481"
server_key_version: "encryptionkey@0"
I20260620 03:04:03.075186 20978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 03:04:03.078612 20993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:03.079949 20978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 03:04:03.080128 20978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "192db2208a3c49c7b7355792f3a8cb27"
format_stamp: "Formatted at 2026-06-20 03:04:03 on dist-test-slave-f1j0"
server_key: "bf74f883b43b71672779a38e609fbfcb"
server_key_iv: "aaa7de43ef00d55988c210d4f7243481"
server_key_version: "encryptionkey@0"
I20260620 03:04:03.080312 20978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:03.086030 20978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:03.090327 20978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:03.090619 20978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:03.109493 20978 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38971
I20260620 03:04:03.109555 21044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38971 every 8 connection(s)
I20260620 03:04:03.110992 20978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:04:03.114919 21045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:03.120029 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 20978
I20260620 03:04:03.120218 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:04:03.120627 388 external_mini_cluster.cc:1428] Setting key 955ed2a99e115b4d0d5389a44ab595e1
I20260620 03:04:03.125972 21045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27: Bootstrap starting.
I20260620 03:04:03.128688 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:03.122474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56649 (local address 127.0.0.1:38971)
0620 03:04:03.123302 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:03.123313 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:03.123361 (+ 48us) server_negotiation.cc:407] Connection header received
0620 03:04:03.125154 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:03.125179 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:03.125537 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:03.126036 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:03.127080 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:03.127092 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:03.127105 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:03.127124 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:03.127175 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:03.127194 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:03.127434 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:03.127584 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 03:04:03.127773 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":352,"threads_started":1}
I20260620 03:04:03.129527 21045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:03.130959 21045 log.cc:824] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:03.132668 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:04:03.134099 21045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27: No bootstrap required, opened a new log
I20260620 03:04:03.140195 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:03.137140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36783 (local address 127.0.0.1:38971)
0620 03:04:03.137337 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:03.137348 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:03.137804 (+ 456us) server_negotiation.cc:407] Connection header received
0620 03:04:03.137930 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:03.137935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:03.138001 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:03.138120 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:03.139122 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:03.139128 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:03.139132 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:03.139150 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:03.139195 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:03.139213 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:03.139343 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:03.139809 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 03:04:03.139873 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 03:04:03.144116 21045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "192db2208a3c49c7b7355792f3a8cb27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38971 } }
I20260620 03:04:03.144527 21045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:04:03.144641 21045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 192db2208a3c49c7b7355792f3a8cb27, State: Initialized, Role: FOLLOWER
I20260620 03:04:03.145268 21045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "192db2208a3c49c7b7355792f3a8cb27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38971 } }
I20260620 03:04:03.145516 21045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:04:03.145677 21045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:04:03.145864 21045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:03.148195 21045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "192db2208a3c49c7b7355792f3a8cb27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38971 } }
I20260620 03:04:03.148747 21045 leader_election.cc:302] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 192db2208a3c49c7b7355792f3a8cb27; no voters:
I20260620 03:04:03.149356 21045 leader_election.cc:288] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:04:03.149528 21055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:04:03.150192 21055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27 [term 1 LEADER]: Becoming Leader. State: Replica: 192db2208a3c49c7b7355792f3a8cb27, State: Running, Role: LEADER
I20260620 03:04:03.150748 21055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "192db2208a3c49c7b7355792f3a8cb27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38971 } }
I20260620 03:04:03.152668 21056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 192db2208a3c49c7b7355792f3a8cb27. Latest consensus state: current_term: 1 leader_uuid: "192db2208a3c49c7b7355792f3a8cb27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "192db2208a3c49c7b7355792f3a8cb27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38971 } } }
I20260620 03:04:03.153112 21056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:03.153404 21057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "192db2208a3c49c7b7355792f3a8cb27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "192db2208a3c49c7b7355792f3a8cb27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38971 } } }
I20260620 03:04:03.153810 21057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:03.154351 21045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:03.156476 21059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:04:03.162160 21059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:04:03.171221 21059 catalog_manager.cc:1347] Generated new cluster ID: 66ec5b7890f14baa965c6b5771ef3390
I20260620 03:04:03.171347 21059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:04:03.188064 21059 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:04:03.188503 21059 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:04:03.197368 21059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 192db2208a3c49c7b7355792f3a8cb27: Generated new TSK 0
I20260620 03:04:03.198411 21059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:04:03.214224 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 20978
W20260620 03:04:03.215443 21040 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56649
W20260620 03:04:03.216158 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 20978: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:04:03.216310 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 20978
2026-06-20T03:04:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1883 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:04:03 dist-test-slave-f1j0 krb5kdc[21076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:04:03 dist-test-slave-f1j0 krb5kdc[21076](info): set up 2 sockets
Jun 20 03:04:03 dist-test-slave-f1j0 krb5kdc[21076](info): commencing operation
krb5kdc: starting...
W20260620 03:04:04.313997 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:04:04 dist-test-slave-f1j0 krb5kdc[21076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:04:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:04:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:04:04.569466 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45963
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:04:04.894389 21089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:04.894706 21089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:04.894901 21089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:04.902151 21089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:04.902288 21089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:04.902396 21089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:04.902499 21089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:04.902590 21089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:04.912288 21089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:04.913595 21089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:04.914922 21089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:04.926582 21094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:04.926626 21095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:04.926539 21097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:04.927256 21089 server_base.cc:1034] running on GCE node
I20260620 03:04:04.927970 21089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:04.929271 21089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:04.930495 21089 hybrid_clock.cc:648] HybridClock initialized: now 1781924644930449 us; error 51 us; skew 500 ppm
I20260620 03:04:04.930832 21089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:04 dist-test-slave-f1j0 krb5kdc[21076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:04.933830 21089 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:04:04.935253 21089 webserver.cc:460] Webserver started at http://127.0.0.1:35609/ using document root <none> and password file <none>
I20260620 03:04:04.935917 21089 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:04.936059 21089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:04.936357 21089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:04.939265 21089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "e613604947b54ed6be4fbfaed9c2431b"
format_stamp: "Formatted at 2026-06-20 03:04:04 on dist-test-slave-f1j0"
server_key: "2314c767288e24f044fa45ffa8a38d4e"
server_key_iv: "cf307f77f21efe928349b3cdb4951877"
server_key_version: "encryptionkey@0"
I20260620 03:04:04.939996 21089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "e613604947b54ed6be4fbfaed9c2431b"
format_stamp: "Formatted at 2026-06-20 03:04:04 on dist-test-slave-f1j0"
server_key: "2314c767288e24f044fa45ffa8a38d4e"
server_key_iv: "cf307f77f21efe928349b3cdb4951877"
server_key_version: "encryptionkey@0"
I20260620 03:04:04.944901 21089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 03:04:04.948195 21104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:04.949452 21089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 03:04:04.949677 21089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "e613604947b54ed6be4fbfaed9c2431b"
format_stamp: "Formatted at 2026-06-20 03:04:04 on dist-test-slave-f1j0"
server_key: "2314c767288e24f044fa45ffa8a38d4e"
server_key_iv: "cf307f77f21efe928349b3cdb4951877"
server_key_version: "encryptionkey@0"
I20260620 03:04:04.949885 21089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:04.973482 21089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:04.977661 21089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:04.977948 21089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:04.996593 21089 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45963
I20260620 03:04:04.996635 21155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45963 every 8 connection(s)
I20260620 03:04:04.998041 21089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:04:05.001789 21156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:05.002025 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 21089
I20260620 03:04:05.002219 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:04:05.002786 388 external_mini_cluster.cc:1428] Setting key 093eed4d02a40eda6ed06fd58289a764
I20260620 03:04:05.017277 21156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b: Bootstrap starting.
Jun 20 03:04:05 dist-test-slave-f1j0 krb5kdc[21076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 03:04:05.020860 21156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:05.022001 21156 log.cc:824] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:05.024767 21156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b: No bootstrap required, opened a new log
I20260620 03:04:05.026274 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:05.004625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42429 (local address 127.0.0.1:45963)
0620 03:04:05.005457 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:05.005468 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:05.005509 (+ 41us) server_negotiation.cc:407] Connection header received
0620 03:04:05.007084 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:05.007106 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:05.007518 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:05.007866 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:05.009871 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:05.011184 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:05.013022 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:05.013844 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:05.013986 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:05.018967 (+ 4981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:05.018975 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:05.018990 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:05.019034 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:05.022252 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:05.022993 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:05.023002 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:05.023009 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:05.023186 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:05.023581 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:05.023588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:05.023594 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:05.024093 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:05.024399 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:05.024832 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 03:04:05.025256 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":405,"threads_started":1}
I20260620 03:04:05.029951 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:04:05.033021 21156 raft_consensus.cc:348] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e613604947b54ed6be4fbfaed9c2431b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45963 } }
I20260620 03:04:05.033387 21156 raft_consensus.cc:374] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:04:05.033499 21156 raft_consensus.cc:729] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e613604947b54ed6be4fbfaed9c2431b, State: Initialized, Role: FOLLOWER
I20260620 03:04:05.034157 21156 consensus_queue.cc:260] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e613604947b54ed6be4fbfaed9c2431b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45963 } }
I20260620 03:04:05.034438 21156 raft_consensus.cc:388] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:04:05.034579 21156 raft_consensus.cc:482] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:04:05.034765 21156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:05.037370 21156 raft_consensus.cc:504] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e613604947b54ed6be4fbfaed9c2431b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45963 } }
I20260620 03:04:05.037941 21156 leader_election.cc:302] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e613604947b54ed6be4fbfaed9c2431b; no voters:
I20260620 03:04:05.038546 21156 leader_election.cc:288] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:04:05.038681 21166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:04:05.039373 21166 raft_consensus.cc:686] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b [term 1 LEADER]: Becoming Leader. State: Replica: e613604947b54ed6be4fbfaed9c2431b, State: Running, Role: LEADER
I20260620 03:04:05.039913 21166 consensus_queue.cc:237] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e613604947b54ed6be4fbfaed9c2431b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45963 } }
I20260620 03:04:05.043198 21168 sys_catalog.cc:455] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e613604947b54ed6be4fbfaed9c2431b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e613604947b54ed6be4fbfaed9c2431b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45963 } } }
I20260620 03:04:05.043548 21168 sys_catalog.cc:458] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:05.041913 21166 sys_catalog.cc:455] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e613604947b54ed6be4fbfaed9c2431b. Latest consensus state: current_term: 1 leader_uuid: "e613604947b54ed6be4fbfaed9c2431b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e613604947b54ed6be4fbfaed9c2431b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45963 } } }
I20260620 03:04:05.044023 21166 sys_catalog.cc:458] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:05.047179 21156 sys_catalog.cc:564] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:05.047397 21170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:04:05.052098 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:05.034193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57489 (local address 127.0.0.1:45963)
0620 03:04:05.034388 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:05.034394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:05.034839 (+ 445us) server_negotiation.cc:407] Connection header received
0620 03:04:05.035012 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:05.035017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:05.035099 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:05.035206 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:05.036506 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:05.037607 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:05.039302 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:05.039990 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:05.040105 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:05.043605 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:05.043611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:05.043616 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:05.043656 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:05.047887 (+ 4231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:05.048772 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:05.048777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:05.048780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:05.048942 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:05.049476 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:05.049485 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:05.049491 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:05.050067 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:05.050383 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:05.051187 (+ 804us) server_negotiation.cc:299] Negotiation successful
0620 03:04:05.051516 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 03:04:05.052770 21170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:04:05.066516 21170 catalog_manager.cc:1347] Generated new cluster ID: de9763ae63d447c48364563e692df855
I20260620 03:04:05.066644 21170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:04:05.075726 21170 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:04:05.077555 21170 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:04:05.091022 21170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e613604947b54ed6be4fbfaed9c2431b: Generated new TSK 0
I20260620 03:04:05.091879 21170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:04:05.105288 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 21089
I20260620 03:04:05.106462 21151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:04:05.200326 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 21089
2026-06-20T03:04:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1984 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:04:05 dist-test-slave-f1j0 krb5kdc[21188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:04:05 dist-test-slave-f1j0 krb5kdc[21188](info): set up 2 sockets
Jun 20 03:04:05 dist-test-slave-f1j0 krb5kdc[21188](info): commencing operation
krb5kdc: starting...
W20260620 03:04:06.285970 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:04:06 dist-test-slave-f1j0 krb5kdc[21188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:04:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:04:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:04:06.540709 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:04:06.872218 21201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:06.872576 21201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:06.872742 21201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:06.880525 21201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:06.880677 21201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:06.880787 21201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:06.880878 21201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:06.880944 21201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:06.890908 21201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:06.892247 21201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:06.893590 21201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:06.905498 21206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:06.905536 21207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:06.905457 21209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:06.906884 21201 server_base.cc:1034] running on GCE node
I20260620 03:04:06.907721 21201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:06.909124 21201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:06.910324 21201 hybrid_clock.cc:648] HybridClock initialized: now 1781924646910287 us; error 35 us; skew 500 ppm
I20260620 03:04:06.910652 21201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:06 dist-test-slave-f1j0 krb5kdc[21188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:06.913709 21201 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:04:06.915208 21201 webserver.cc:460] Webserver started at http://127.0.0.1:37719/ using document root <none> and password file <none>
I20260620 03:04:06.915884 21201 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:06.916015 21201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:06.916327 21201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:06.919308 21201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "b9bfe2aa34d04bc68171067a274b5ce4"
format_stamp: "Formatted at 2026-06-20 03:04:06 on dist-test-slave-f1j0"
server_key: "a8d3bb6a37d28b14a3c22fde5f4f42bf"
server_key_iv: "1d33583cd3e8134b22b8fe4166014282"
server_key_version: "encryptionkey@0"
I20260620 03:04:06.920095 21201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "b9bfe2aa34d04bc68171067a274b5ce4"
format_stamp: "Formatted at 2026-06-20 03:04:06 on dist-test-slave-f1j0"
server_key: "a8d3bb6a37d28b14a3c22fde5f4f42bf"
server_key_iv: "1d33583cd3e8134b22b8fe4166014282"
server_key_version: "encryptionkey@0"
I20260620 03:04:06.925264 21201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 03:04:06.928711 21216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:06.930142 21201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 03:04:06.930322 21201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "b9bfe2aa34d04bc68171067a274b5ce4"
format_stamp: "Formatted at 2026-06-20 03:04:06 on dist-test-slave-f1j0"
server_key: "a8d3bb6a37d28b14a3c22fde5f4f42bf"
server_key_iv: "1d33583cd3e8134b22b8fe4166014282"
server_key_version: "encryptionkey@0"
I20260620 03:04:06.930516 21201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:06.956753 21201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:06.960817 21201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:06.961123 21201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:06.979805 21201 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44989
I20260620 03:04:06.979851 21267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44989 every 8 connection(s)
I20260620 03:04:06.981202 21201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:04:06.984993 21268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:06.985708 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 21201
I20260620 03:04:06.985913 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:04:06.986338 388 external_mini_cluster.cc:1428] Setting key 82f991401df8a13e89e805f475656895
I20260620 03:04:06.998505 21268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4: Bootstrap starting.
I20260620 03:04:06.999915 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:06.988057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53293 (local address 127.0.0.1:44989)
0620 03:04:06.988819 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:06.988830 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:06.988874 (+ 44us) server_negotiation.cc:407] Connection header received
0620 03:04:06.990954 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:06.990976 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:06.991395 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:06.991747 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:06.992299 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:06.993526 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:06.995457 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:06.996174 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:06.996302 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:06.997630 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:06.997647 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:06.997661 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:06.998197 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:06.998272 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:06.998296 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:06.998380 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:06.998679 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 03:04:06.998978 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":352,"threads_started":1}
I20260620 03:04:07.001785 21268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:07.003033 21268 log.cc:824] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:07.003719 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:04:07.005816 21268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4: No bootstrap required, opened a new log
I20260620 03:04:07.014192 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:07.008026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37967 (local address 127.0.0.1:44989)
0620 03:04:07.008228 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:07.008236 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:07.008567 (+ 331us) server_negotiation.cc:407] Connection header received
0620 03:04:07.008779 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:07.008783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:07.008842 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:07.008951 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:07.009495 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:07.010299 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:07.011668 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:07.012137 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:07.012290 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:07.013381 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:07.013387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:07.013390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:07.013407 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:07.013444 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:07.013458 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:07.013558 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:07.013726 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 03:04:07.013817 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 03:04:07.013870 21268 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bfe2aa34d04bc68171067a274b5ce4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260620 03:04:07.014446 21268 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:04:07.014564 21268 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9bfe2aa34d04bc68171067a274b5ce4, State: Initialized, Role: FOLLOWER
I20260620 03:04:07.015250 21268 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bfe2aa34d04bc68171067a274b5ce4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260620 03:04:07.015520 21268 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:04:07.015689 21268 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:04:07.015863 21268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:07.018179 21268 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bfe2aa34d04bc68171067a274b5ce4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260620 03:04:07.018708 21268 leader_election.cc:302] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9bfe2aa34d04bc68171067a274b5ce4; no voters:
I20260620 03:04:07.019433 21268 leader_election.cc:288] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:04:07.019562 21278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:04:07.020216 21278 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4 [term 1 LEADER]: Becoming Leader. State: Replica: b9bfe2aa34d04bc68171067a274b5ce4, State: Running, Role: LEADER
I20260620 03:04:07.020975 21278 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bfe2aa34d04bc68171067a274b5ce4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260620 03:04:07.023438 21278 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9bfe2aa34d04bc68171067a274b5ce4. Latest consensus state: current_term: 1 leader_uuid: "b9bfe2aa34d04bc68171067a274b5ce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bfe2aa34d04bc68171067a274b5ce4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } } }
I20260620 03:04:07.023895 21278 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:07.024232 21280 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9bfe2aa34d04bc68171067a274b5ce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bfe2aa34d04bc68171067a274b5ce4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } } }
I20260620 03:04:07.024567 21280 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:07.026899 21268 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:07.027240 21282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:04:07.032831 21282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:04:07.043641 21282 catalog_manager.cc:1347] Generated new cluster ID: 011df4384e3141bd99898048d048e330
I20260620 03:04:07.043766 21282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:04:07.061499 21282 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:04:07.062825 21282 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:04:07.072733 21282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9bfe2aa34d04bc68171067a274b5ce4: Generated new TSK 0
I20260620 03:04:07.073539 21282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:04:07.082747 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 21201
W20260620 03:04:07.083853 21263 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53293
W20260620 03:04:07.084566 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 21201: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:04:07.084728 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 21201
2026-06-20T03:04:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1884 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:04:07 dist-test-slave-f1j0 krb5kdc[21299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:04:07 dist-test-slave-f1j0 krb5kdc[21299](info): set up 2 sockets
Jun 20 03:04:07 dist-test-slave-f1j0 krb5kdc[21299](info): commencing operation
krb5kdc: starting...
W20260620 03:04:08.193897 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:04:08 dist-test-slave-f1j0 krb5kdc[21299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:04:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:04:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:04:08.453137 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32891
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:04:08.782796 21312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:08.783103 21312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:08.783272 21312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:08.790725 21312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:08.790848 21312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:08.790956 21312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:08.791065 21312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:08.791143 21312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:08.801295 21312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:08.802706 21312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:08.804044 21312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:08.815970 21317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:08.815997 21320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:08.815918 21318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:08.816871 21312 server_base.cc:1034] running on GCE node
I20260620 03:04:08.817672 21312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:08.819062 21312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:08.820286 21312 hybrid_clock.cc:648] HybridClock initialized: now 1781924648820217 us; error 51 us; skew 500 ppm
I20260620 03:04:08.820727 21312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:08 dist-test-slave-f1j0 krb5kdc[21299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:08.827296 21312 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:04:08.828780 21312 webserver.cc:460] Webserver started at http://127.0.0.1:36199/ using document root <none> and password file <none>
I20260620 03:04:08.829417 21312 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:08.829550 21312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:08.829874 21312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:08.832749 21312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "64cb6f2d382840ed9f0c7c3d36133797"
format_stamp: "Formatted at 2026-06-20 03:04:08 on dist-test-slave-f1j0"
server_key: "79c946dd7da6fbd73d55e146cb860294"
server_key_iv: "cb23dd3ddf33fb327758cdf6d6ecef6d"
server_key_version: "encryptionkey@0"
I20260620 03:04:08.833424 21312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "64cb6f2d382840ed9f0c7c3d36133797"
format_stamp: "Formatted at 2026-06-20 03:04:08 on dist-test-slave-f1j0"
server_key: "79c946dd7da6fbd73d55e146cb860294"
server_key_iv: "cb23dd3ddf33fb327758cdf6d6ecef6d"
server_key_version: "encryptionkey@0"
I20260620 03:04:08.838471 21312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.002s
I20260620 03:04:08.841766 21327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:08.843117 21312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:04:08.843297 21312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "64cb6f2d382840ed9f0c7c3d36133797"
format_stamp: "Formatted at 2026-06-20 03:04:08 on dist-test-slave-f1j0"
server_key: "79c946dd7da6fbd73d55e146cb860294"
server_key_iv: "cb23dd3ddf33fb327758cdf6d6ecef6d"
server_key_version: "encryptionkey@0"
I20260620 03:04:08.843492 21312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:08.848973 21312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:08.852892 21312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:08.853173 21312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:08.871157 21312 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32891
I20260620 03:04:08.871206 21378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32891 every 8 connection(s)
I20260620 03:04:08.872643 21312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:04:08.876530 21379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:08.878070 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 21312
I20260620 03:04:08.878249 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:04:08.878719 388 external_mini_cluster.cc:1428] Setting key 53e36cf7578cd1fd177fcb6ce1ac28be
I20260620 03:04:08.886931 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:08.880571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57211 (local address 127.0.0.1:32891)
0620 03:04:08.881370 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:08.881386 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:08.881446 (+ 60us) server_negotiation.cc:407] Connection header received
0620 03:04:08.883230 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:08.883257 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:08.883711 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:08.884153 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:08.885088 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:08.885095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:08.885113 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:08.885128 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:08.885188 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:08.885216 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:08.885418 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:08.885555 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 03:04:08.885773 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":328,"threads_started":1}
I20260620 03:04:08.890493 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:04:08.890663 21379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797: Bootstrap starting.
I20260620 03:04:08.893766 21379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:08.894953 21379 log.cc:824] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:08.897070 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:08.894715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36575 (local address 127.0.0.1:32891)
0620 03:04:08.894886 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:08.894893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:08.895176 (+ 283us) server_negotiation.cc:407] Connection header received
0620 03:04:08.895308 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:08.895313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:08.895375 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:08.895487 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:08.896454 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:08.896459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:08.896462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:08.896478 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:08.896518 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:08.896535 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:08.896678 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:08.896749 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 03:04:08.896804 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 03:04:08.898141 21379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797: No bootstrap required, opened a new log
I20260620 03:04:08.905514 21379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb6f2d382840ed9f0c7c3d36133797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32891 } }
I20260620 03:04:08.905946 21379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:04:08.906064 21379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64cb6f2d382840ed9f0c7c3d36133797, State: Initialized, Role: FOLLOWER
I20260620 03:04:08.906594 21379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb6f2d382840ed9f0c7c3d36133797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32891 } }
I20260620 03:04:08.906872 21379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:04:08.907025 21379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:04:08.907183 21379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:08.909708 21379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb6f2d382840ed9f0c7c3d36133797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32891 } }
I20260620 03:04:08.910172 21379 leader_election.cc:302] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64cb6f2d382840ed9f0c7c3d36133797; no voters:
I20260620 03:04:08.910686 21379 leader_election.cc:288] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:04:08.910831 21389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:04:08.911545 21389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797 [term 1 LEADER]: Becoming Leader. State: Replica: 64cb6f2d382840ed9f0c7c3d36133797, State: Running, Role: LEADER
I20260620 03:04:08.912110 21389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb6f2d382840ed9f0c7c3d36133797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32891 } }
I20260620 03:04:08.912840 21379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:08.915107 21391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64cb6f2d382840ed9f0c7c3d36133797. Latest consensus state: current_term: 1 leader_uuid: "64cb6f2d382840ed9f0c7c3d36133797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb6f2d382840ed9f0c7c3d36133797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32891 } } }
I20260620 03:04:08.915558 21391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:08.917832 21390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64cb6f2d382840ed9f0c7c3d36133797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb6f2d382840ed9f0c7c3d36133797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32891 } } }
I20260620 03:04:08.918493 21390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:08.918938 21396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:04:08.924876 21396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:04:08.932837 21396 catalog_manager.cc:1347] Generated new cluster ID: fc3694a97e604e0fa61a38b2c94fc408
I20260620 03:04:08.932935 21396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:04:08.942160 21396 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:04:08.942632 21396 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:04:08.956447 21396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64cb6f2d382840ed9f0c7c3d36133797: Generated new TSK 0
I20260620 03:04:08.957469 21396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:04:08.964700 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 21312
W20260620 03:04:08.965811 21374 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57211
W20260620 03:04:08.966511 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 21312: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:04:08.966665 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 21312
2026-06-20T03:04:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1881 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:04:09 dist-test-slave-f1j0 krb5kdc[21410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:04:09 dist-test-slave-f1j0 krb5kdc[21410](info): set up 2 sockets
Jun 20 03:04:09 dist-test-slave-f1j0 krb5kdc[21410](info): commencing operation
krb5kdc: starting...
W20260620 03:04:10.066013 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:04:10 dist-test-slave-f1j0 krb5kdc[21410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:04:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:04:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:04:10.332237 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39721
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:04:10.661381 21423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:10.661764 21423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:10.661863 21423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:10.669126 21423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:10.669238 21423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:10.669284 21423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:10.669337 21423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:10.669381 21423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:10.680199 21423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:10.681504 21423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:10.682889 21423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:10.694941 21428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:10.694880 21429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:10.694911 21431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:10.695696 21423 server_base.cc:1034] running on GCE node
I20260620 03:04:10.696504 21423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:10.697952 21423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:10.699187 21423 hybrid_clock.cc:648] HybridClock initialized: now 1781924650699134 us; error 40 us; skew 500 ppm
I20260620 03:04:10.699532 21423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:10 dist-test-slave-f1j0 krb5kdc[21410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:10.702610 21423 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:04:10.704145 21423 webserver.cc:460] Webserver started at http://127.0.0.1:34271/ using document root <none> and password file <none>
I20260620 03:04:10.704829 21423 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:10.704977 21423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:10.705320 21423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:10.708287 21423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "9b07997fb8dc4542bcc1dda0440239ec"
format_stamp: "Formatted at 2026-06-20 03:04:10 on dist-test-slave-f1j0"
server_key: "1475f3f3dc2493264b8a47585a52a04c"
server_key_iv: "fb1d11ea753a1b9a8c4dbb8b1030ed5f"
server_key_version: "encryptionkey@0"
I20260620 03:04:10.709044 21423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "9b07997fb8dc4542bcc1dda0440239ec"
format_stamp: "Formatted at 2026-06-20 03:04:10 on dist-test-slave-f1j0"
server_key: "1475f3f3dc2493264b8a47585a52a04c"
server_key_iv: "fb1d11ea753a1b9a8c4dbb8b1030ed5f"
server_key_version: "encryptionkey@0"
I20260620 03:04:10.714093 21423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 03:04:10.717509 21438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:10.718935 21423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:04:10.719125 21423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "9b07997fb8dc4542bcc1dda0440239ec"
format_stamp: "Formatted at 2026-06-20 03:04:10 on dist-test-slave-f1j0"
server_key: "1475f3f3dc2493264b8a47585a52a04c"
server_key_iv: "fb1d11ea753a1b9a8c4dbb8b1030ed5f"
server_key_version: "encryptionkey@0"
I20260620 03:04:10.719326 21423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:10.740376 21423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:10.744478 21423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:10.744779 21423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:10.763233 21423 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39721
I20260620 03:04:10.763266 21489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39721 every 8 connection(s)
I20260620 03:04:10.764582 21423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:04:10.766353 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 21423
I20260620 03:04:10.766558 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:04:10.766983 388 external_mini_cluster.cc:1428] Setting key 3e5fd9d9f60eb90c61a06d7270788a66
I20260620 03:04:10.768833 21490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 03:04:10 dist-test-slave-f1j0 krb5kdc[21410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 03:04:10.782358 21490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec: Bootstrap starting.
I20260620 03:04:10.786553 21490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:10.787760 21490 log.cc:824] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:10.791241 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:10.768702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:39591 (local address 127.0.0.1:39721)
0620 03:04:10.769437 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:10.769448 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:10.769500 (+ 52us) server_negotiation.cc:407] Connection header received
0620 03:04:10.771159 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:10.771180 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:10.771563 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:10.771885 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:10.773697 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:10.775012 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:10.776804 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:10.777432 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:10.777550 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:10.782467 (+ 4917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:10.782478 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:10.782497 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:10.782546 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:10.786384 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:10.787178 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:10.787186 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:10.787194 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:10.787410 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:10.787906 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:10.787911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:10.787913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:10.788606 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:10.788857 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:10.789536 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 03:04:10.789900 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":344,"threads_started":1}
I20260620 03:04:10.791374 21490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec: No bootstrap required, opened a new log
I20260620 03:04:10.795261 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:04:10.799418 21490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b07997fb8dc4542bcc1dda0440239ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39721 } }
I20260620 03:04:10.799873 21490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:04:10.800045 21490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b07997fb8dc4542bcc1dda0440239ec, State: Initialized, Role: FOLLOWER
I20260620 03:04:10.800771 21490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b07997fb8dc4542bcc1dda0440239ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39721 } }
I20260620 03:04:10.801080 21490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:04:10.801224 21490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:04:10.801404 21490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:10.804394 21490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b07997fb8dc4542bcc1dda0440239ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39721 } }
I20260620 03:04:10.804967 21490 leader_election.cc:302] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b07997fb8dc4542bcc1dda0440239ec; no voters:
I20260620 03:04:10.805799 21490 leader_election.cc:288] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:04:10.805909 21500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:04:10.808808 21500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec [term 1 LEADER]: Becoming Leader. State: Replica: 9b07997fb8dc4542bcc1dda0440239ec, State: Running, Role: LEADER
I20260620 03:04:10.809007 21490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:10.809521 21500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b07997fb8dc4542bcc1dda0440239ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39721 } }
I20260620 03:04:10.815908 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:10.799751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:40885 (local address 127.0.0.1:39721)
0620 03:04:10.799990 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:10.799998 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:10.800426 (+ 428us) server_negotiation.cc:407] Connection header received
0620 03:04:10.800590 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:10.800594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:10.800656 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:10.800735 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:10.802625 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:10.803454 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:10.805354 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:10.806009 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:10.806165 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:10.808602 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:10.808611 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:10.808618 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:10.808663 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:10.812696 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:10.813761 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:10.813765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:10.813767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:10.813933 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:10.814389 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:10.814395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:10.814398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:10.814688 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:10.815018 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:10.815381 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 03:04:10.815562 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 03:04:10.821664 21501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b07997fb8dc4542bcc1dda0440239ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b07997fb8dc4542bcc1dda0440239ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39721 } } }
I20260620 03:04:10.821709 21502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b07997fb8dc4542bcc1dda0440239ec. Latest consensus state: current_term: 1 leader_uuid: "9b07997fb8dc4542bcc1dda0440239ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b07997fb8dc4542bcc1dda0440239ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39721 } } }
I20260620 03:04:10.822227 21501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:10.822314 21502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:10.831817 21514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:04:10.836030 21514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:04:10.844465 21514 catalog_manager.cc:1347] Generated new cluster ID: a13a29b3be7b4d6686b3c9be6d852c6b
I20260620 03:04:10.844592 21514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:04:10.861918 21514 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:04:10.863221 21514 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:04:10.873493 21514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b07997fb8dc4542bcc1dda0440239ec: Generated new TSK 0
I20260620 03:04:10.874634 21514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:04:10.889036 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 21423
I20260620 03:04:10.890323 21485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:04:10.982746 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 21423
W20260620 03:04:11.000139 21408 connection.cc:535] client connection to 127.0.0.1:39721 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39721): Connection reset by peer (error 104)
2026-06-20T03:04:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2016 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:04:11 dist-test-slave-f1j0 krb5kdc[21523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:04:11 dist-test-slave-f1j0 krb5kdc[21523](info): set up 2 sockets
Jun 20 03:04:11 dist-test-slave-f1j0 krb5kdc[21523](info): commencing operation
krb5kdc: starting...
W20260620 03:04:12.089955 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:04:12 dist-test-slave-f1j0 krb5kdc[21523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:04:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:04:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:04:12.348352 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34145
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:04:12.679951 21536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:12.680321 21536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:12.680497 21536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:12.687930 21536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:12.688099 21536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:12.688208 21536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:12.688316 21536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:12.688400 21536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:12.698827 21536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:12.700248 21536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:12.701546 21536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:12.713120 21541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:12.713057 21542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:12.713178 21544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:12.714340 21536 server_base.cc:1034] running on GCE node
I20260620 03:04:12.715248 21536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:12.716751 21536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:12.717975 21536 hybrid_clock.cc:648] HybridClock initialized: now 1781924652717929 us; error 44 us; skew 500 ppm
I20260620 03:04:12.718318 21536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:12 dist-test-slave-f1j0 krb5kdc[21523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:12.721397 21536 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:04:12.723098 21536 webserver.cc:460] Webserver started at http://127.0.0.1:37727/ using document root <none> and password file <none>
I20260620 03:04:12.723810 21536 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:12.723961 21536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:12.724288 21536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:12.727389 21536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "b48dd903a99e4036ae0d7904bc0d94ec"
format_stamp: "Formatted at 2026-06-20 03:04:12 on dist-test-slave-f1j0"
server_key: "1d64c07cfa3814ed22f767c2df655f73"
server_key_iv: "646f1426544adfb88ef11780cd43d86a"
server_key_version: "encryptionkey@0"
I20260620 03:04:12.728178 21536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "b48dd903a99e4036ae0d7904bc0d94ec"
format_stamp: "Formatted at 2026-06-20 03:04:12 on dist-test-slave-f1j0"
server_key: "1d64c07cfa3814ed22f767c2df655f73"
server_key_iv: "646f1426544adfb88ef11780cd43d86a"
server_key_version: "encryptionkey@0"
I20260620 03:04:12.733556 21536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 03:04:12.737149 21551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:12.738559 21536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:04:12.738787 21536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "b48dd903a99e4036ae0d7904bc0d94ec"
format_stamp: "Formatted at 2026-06-20 03:04:12 on dist-test-slave-f1j0"
server_key: "1d64c07cfa3814ed22f767c2df655f73"
server_key_iv: "646f1426544adfb88ef11780cd43d86a"
server_key_version: "encryptionkey@0"
I20260620 03:04:12.739003 21536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:12.766093 21536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:12.770442 21536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:12.770771 21536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:12.790189 21536 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34145
I20260620 03:04:12.790226 21602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34145 every 8 connection(s)
I20260620 03:04:12.791644 21536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:04:12.793910 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 21536
I20260620 03:04:12.794098 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:04:12.794556 388 external_mini_cluster.cc:1428] Setting key 374eea56d0123ec708dd4de8f54f7559
I20260620 03:04:12.795887 21603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:12.808025 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:12.796452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:40865 (local address 127.0.0.1:34145)
0620 03:04:12.797267 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:12.797278 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:12.797319 (+ 41us) server_negotiation.cc:407] Connection header received
0620 03:04:12.798936 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:12.798957 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:12.799373 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:12.799703 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:12.800318 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:12.801741 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:12.803610 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:12.804258 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:12.804377 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:12.805754 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:12.805765 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:12.805778 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:12.806338 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:12.806410 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:12.806436 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:12.806554 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:12.806863 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 03:04:12.807063 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":368,"threads_started":1}
I20260620 03:04:12.809196 21603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec: Bootstrap starting.
I20260620 03:04:12.811786 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:04:12.812557 21603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:12.813506 21603 log.cc:824] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:12.816802 21603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec: No bootstrap required, opened a new log
I20260620 03:04:12.822147 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:12.816056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:39081 (local address 127.0.0.1:34145)
0620 03:04:12.816245 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:12.816254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:12.816640 (+ 386us) server_negotiation.cc:407] Connection header received
0620 03:04:12.816869 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:12.816874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:12.816951 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:12.817051 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:12.817549 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:12.818337 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:12.819627 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:12.820083 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:12.820250 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:12.821359 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:12.821364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:12.821367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:12.821383 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:12.821437 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:12.821452 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:12.821575 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:12.821808 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 03:04:12.821866 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:04:12.824954 21603 raft_consensus.cc:348] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b48dd903a99e4036ae0d7904bc0d94ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34145 } }
I20260620 03:04:12.825330 21603 raft_consensus.cc:374] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:04:12.825445 21603 raft_consensus.cc:729] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b48dd903a99e4036ae0d7904bc0d94ec, State: Initialized, Role: FOLLOWER
I20260620 03:04:12.826059 21603 consensus_queue.cc:260] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b48dd903a99e4036ae0d7904bc0d94ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34145 } }
I20260620 03:04:12.826285 21603 raft_consensus.cc:388] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:04:12.826442 21603 raft_consensus.cc:482] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:04:12.826594 21603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:12.829023 21603 raft_consensus.cc:504] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b48dd903a99e4036ae0d7904bc0d94ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34145 } }
I20260620 03:04:12.829629 21603 leader_election.cc:302] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b48dd903a99e4036ae0d7904bc0d94ec; no voters:
I20260620 03:04:12.830199 21603 leader_election.cc:288] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:04:12.830323 21613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:04:12.831024 21613 raft_consensus.cc:686] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec [term 1 LEADER]: Becoming Leader. State: Replica: b48dd903a99e4036ae0d7904bc0d94ec, State: Running, Role: LEADER
I20260620 03:04:12.831576 21613 consensus_queue.cc:237] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b48dd903a99e4036ae0d7904bc0d94ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34145 } }
I20260620 03:04:12.832336 21603 sys_catalog.cc:564] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:12.834064 21615 sys_catalog.cc:455] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader b48dd903a99e4036ae0d7904bc0d94ec. Latest consensus state: current_term: 1 leader_uuid: "b48dd903a99e4036ae0d7904bc0d94ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b48dd903a99e4036ae0d7904bc0d94ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34145 } } }
I20260620 03:04:12.834440 21615 sys_catalog.cc:458] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:12.834764 21614 sys_catalog.cc:455] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b48dd903a99e4036ae0d7904bc0d94ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b48dd903a99e4036ae0d7904bc0d94ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34145 } } }
I20260620 03:04:12.835204 21614 sys_catalog.cc:458] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:12.842721 21618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:04:12.848349 21618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:04:12.856832 21618 catalog_manager.cc:1347] Generated new cluster ID: ea48762e657d41049940efee334e2cfe
I20260620 03:04:12.856947 21618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:04:12.876329 21618 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:04:12.877708 21618 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:04:12.883167 21618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b48dd903a99e4036ae0d7904bc0d94ec: Generated new TSK 0
I20260620 03:04:12.883970 21618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:04:12.895682 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 21536
W20260620 03:04:12.896863 21598 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.99:40865
W20260620 03:04:12.897660 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 21536: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:04:12.897818 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 21536
W20260620 03:04:12.914693 21521 connection.cc:535] client connection to 127.0.0.1:34145 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34145): Connection reset by peer (error 104)
2026-06-20T03:04:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1914 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:04:12 dist-test-slave-f1j0 krb5kdc[21635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:04:12 dist-test-slave-f1j0 krb5kdc[21635](info): set up 2 sockets
Jun 20 03:04:12 dist-test-slave-f1j0 krb5kdc[21635](info): commencing operation
krb5kdc: starting...
W20260620 03:04:13.986032 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:04:14 dist-test-slave-f1j0 krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:04:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:04:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:04:14.245142 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:04:14.578579 21648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:14.578864 21648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:14.578956 21648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:14.586009 21648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:14.586107 21648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:14.586177 21648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:14.586237 21648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:14.586280 21648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:14.595772 21648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:14.596985 21648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:14.598245 21648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:14.610247 21656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:14.610292 21654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:14.610337 21653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:14.611620 21648 server_base.cc:1034] running on GCE node
I20260620 03:04:14.612537 21648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:14.614105 21648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:14.615334 21648 hybrid_clock.cc:648] HybridClock initialized: now 1781924654615296 us; error 26 us; skew 500 ppm
I20260620 03:04:14.615710 21648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:14 dist-test-slave-f1j0 krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:14.622130 21648 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:04:14.623809 21648 webserver.cc:460] Webserver started at http://127.0.0.1:39837/ using document root <none> and password file <none>
I20260620 03:04:14.624538 21648 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:14.624691 21648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:14.625008 21648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:14.628147 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "ad4de5b26769409f83c6ee0825ef6949"
format_stamp: "Formatted at 2026-06-20 03:04:14 on dist-test-slave-f1j0"
server_key: "bdad47015640ed604741615627ee043f"
server_key_iv: "0c4da4decc8860c2b36cdaa87b8ea688"
server_key_version: "encryptionkey@0"
I20260620 03:04:14.628953 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "ad4de5b26769409f83c6ee0825ef6949"
format_stamp: "Formatted at 2026-06-20 03:04:14 on dist-test-slave-f1j0"
server_key: "bdad47015640ed604741615627ee043f"
server_key_iv: "0c4da4decc8860c2b36cdaa87b8ea688"
server_key_version: "encryptionkey@0"
I20260620 03:04:14.634409 21648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 03:04:14.637951 21663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:14.639272 21648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 03:04:14.639456 21648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "ad4de5b26769409f83c6ee0825ef6949"
format_stamp: "Formatted at 2026-06-20 03:04:14 on dist-test-slave-f1j0"
server_key: "bdad47015640ed604741615627ee043f"
server_key_iv: "0c4da4decc8860c2b36cdaa87b8ea688"
server_key_version: "encryptionkey@0"
I20260620 03:04:14.639704 21648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:14.645437 21648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:14.650090 21648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:14.656706 21648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:14.674649 21648 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37305
I20260620 03:04:14.674670 21714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37305 every 8 connection(s)
I20260620 03:04:14.676066 21648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:04:14.679908 21715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:14.680439 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 21648
I20260620 03:04:14.680629 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:04:14.681056 388 external_mini_cluster.cc:1428] Setting key 97876d2b7c6ac74a6d6b4b7c0dc42e15
I20260620 03:04:14.690541 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:14.682921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:34979 (local address 127.0.0.1:37305)
0620 03:04:14.683685 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:14.683700 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:14.683755 (+ 55us) server_negotiation.cc:407] Connection header received
0620 03:04:14.686604 (+ 2849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:14.686637 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:14.687109 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:14.687585 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:14.688539 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:14.688548 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:14.688564 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:14.688579 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:14.688641 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:14.688671 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:14.688882 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:14.689075 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 03:04:14.689283 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":312,"threads_started":1}
I20260620 03:04:14.694383 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:04:14.694906 21715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949: Bootstrap starting.
I20260620 03:04:14.698235 21715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:14.699435 21715 log.cc:824] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:14.701418 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:14.698962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:54965 (local address 127.0.0.1:37305)
0620 03:04:14.699151 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:14.699162 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:14.699523 (+ 361us) server_negotiation.cc:407] Connection header received
0620 03:04:14.699721 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:14.699727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:14.699788 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:14.699893 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:14.700791 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:14.700797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:14.700801 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:14.700819 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:14.700867 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:14.700883 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:14.701000 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:14.701094 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 03:04:14.701157 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 03:04:14.702364 21715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949: No bootstrap required, opened a new log
I20260620 03:04:14.710156 21715 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4de5b26769409f83c6ee0825ef6949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260620 03:04:14.710533 21715 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:04:14.710652 21715 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad4de5b26769409f83c6ee0825ef6949, State: Initialized, Role: FOLLOWER
I20260620 03:04:14.711253 21715 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4de5b26769409f83c6ee0825ef6949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260620 03:04:14.711491 21715 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:04:14.711647 21715 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:04:14.711818 21715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:14.714015 21715 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4de5b26769409f83c6ee0825ef6949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260620 03:04:14.714478 21715 leader_election.cc:302] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad4de5b26769409f83c6ee0825ef6949; no voters:
I20260620 03:04:14.715035 21715 leader_election.cc:288] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:04:14.715219 21725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:04:14.715934 21725 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949 [term 1 LEADER]: Becoming Leader. State: Replica: ad4de5b26769409f83c6ee0825ef6949, State: Running, Role: LEADER
I20260620 03:04:14.716456 21725 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4de5b26769409f83c6ee0825ef6949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260620 03:04:14.717060 21715 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:14.719712 21727 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad4de5b26769409f83c6ee0825ef6949. Latest consensus state: current_term: 1 leader_uuid: "ad4de5b26769409f83c6ee0825ef6949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4de5b26769409f83c6ee0825ef6949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } } }
I20260620 03:04:14.720083 21727 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:14.719801 21726 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad4de5b26769409f83c6ee0825ef6949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4de5b26769409f83c6ee0825ef6949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } } }
I20260620 03:04:14.720201 21726 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:14.722165 21730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:04:14.727213 21730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:04:14.737416 21730 catalog_manager.cc:1347] Generated new cluster ID: 178ac10fde3d48a4b36bc0ac834f6ad5
I20260620 03:04:14.737637 21730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:04:14.747700 21730 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:04:14.748144 21730 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:04:14.756940 21730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad4de5b26769409f83c6ee0825ef6949: Generated new TSK 0
I20260620 03:04:14.757927 21730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:04:14.774768 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 21648
W20260620 03:04:14.775848 21710 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.99:34979
W20260620 03:04:14.776561 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 21648: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:04:14.776715 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 21648
2026-06-20T03:04:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:04:14 dist-test-slave-f1j0 krb5kdc[21746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:04:14 dist-test-slave-f1j0 krb5kdc[21746](info): set up 2 sockets
Jun 20 03:04:14 dist-test-slave-f1j0 krb5kdc[21746](info): commencing operation
krb5kdc: starting...
W20260620 03:04:15.870075 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:04:16 dist-test-slave-f1j0 krb5kdc[21746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:04:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:04:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:04:16.123446 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33157
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:04:16.458969 21759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:16.459317 21759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:16.459486 21759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:16.466640 21759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:16.466797 21759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:16.466897 21759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:16.467002 21759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:16.467082 21759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:16.477133 21759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33157
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:16.478544 21759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:16.479873 21759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:16.492306 21764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:16.492339 21765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:16.492390 21767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:16.493278 21759 server_base.cc:1034] running on GCE node
I20260620 03:04:16.494159 21759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:16.495620 21759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:16.496851 21759 hybrid_clock.cc:648] HybridClock initialized: now 1781924656496784 us; error 54 us; skew 500 ppm
I20260620 03:04:16.497191 21759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:16 dist-test-slave-f1j0 krb5kdc[21746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:16.500504 21759 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:04:16.502192 21759 webserver.cc:460] Webserver started at http://127.0.0.1:35109/ using document root <none> and password file <none>
I20260620 03:04:16.502936 21759 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:16.503072 21759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:16.503404 21759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:16.506596 21759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "2b48823b21864f3e96d4d51170931d5d"
format_stamp: "Formatted at 2026-06-20 03:04:16 on dist-test-slave-f1j0"
server_key: "6b69df01c964c1b5a8880ee4d1c469e9"
server_key_iv: "496da6b0e3208e768888ca4c4823b609"
server_key_version: "encryptionkey@0"
I20260620 03:04:16.507364 21759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "2b48823b21864f3e96d4d51170931d5d"
format_stamp: "Formatted at 2026-06-20 03:04:16 on dist-test-slave-f1j0"
server_key: "6b69df01c964c1b5a8880ee4d1c469e9"
server_key_iv: "496da6b0e3208e768888ca4c4823b609"
server_key_version: "encryptionkey@0"
I20260620 03:04:16.512522 21759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:04:16.515933 21774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:16.517247 21759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 03:04:16.517426 21759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "2b48823b21864f3e96d4d51170931d5d"
format_stamp: "Formatted at 2026-06-20 03:04:16 on dist-test-slave-f1j0"
server_key: "6b69df01c964c1b5a8880ee4d1c469e9"
server_key_iv: "496da6b0e3208e768888ca4c4823b609"
server_key_version: "encryptionkey@0"
I20260620 03:04:16.517693 21759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:16.550203 21759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:16.554100 21759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:16.554337 21759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:16.573040 21759 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33157
I20260620 03:04:16.573064 21825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33157 every 8 connection(s)
I20260620 03:04:16.574530 21759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:04:16.578249 21826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:16.578878 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 21759
I20260620 03:04:16.579075 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:04:16.579520 388 external_mini_cluster.cc:1428] Setting key 4143f52be34eeb9f82a224cefbee43c3
I20260620 03:04:16.592684 21826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d: Bootstrap starting.
Jun 20 03:04:16 dist-test-slave-f1j0 krb5kdc[21746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 03:04:16.595901 21826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:16.597278 21826 log.cc:824] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:16.600618 21826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d: No bootstrap required, opened a new log
I20260620 03:04:16.604216 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:16.581372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:42453 (local address 127.0.0.1:33157)
0620 03:04:16.582159 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:16.582174 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:16.582222 (+ 48us) server_negotiation.cc:407] Connection header received
0620 03:04:16.584818 (+ 2596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:16.584845 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:16.585398 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:16.585856 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:16.587103 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:16.589135 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:16.590484 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:16.591511 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:16.591694 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:16.595767 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:16.595783 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:16.595803 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:16.595856 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:16.599579 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:16.600330 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:16.600337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:16.600345 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:16.600550 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:16.601070 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:16.601077 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:16.601081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:16.601787 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:16.602129 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:16.602430 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 03:04:16.602798 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":323,"threads_started":1}
I20260620 03:04:16.608315 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:04:16.610212 21826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b48823b21864f3e96d4d51170931d5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33157 } }
I20260620 03:04:16.610581 21826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:04:16.610713 21826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b48823b21864f3e96d4d51170931d5d, State: Initialized, Role: FOLLOWER
I20260620 03:04:16.611388 21826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b48823b21864f3e96d4d51170931d5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33157 } }
I20260620 03:04:16.611676 21826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:04:16.611807 21826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:04:16.611989 21826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:16.615343 21826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b48823b21864f3e96d4d51170931d5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33157 } }
I20260620 03:04:16.615971 21826 leader_election.cc:302] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b48823b21864f3e96d4d51170931d5d; no voters:
I20260620 03:04:16.616683 21826 leader_election.cc:288] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:04:16.616776 21836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:04:16.618462 21836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d [term 1 LEADER]: Becoming Leader. State: Replica: 2b48823b21864f3e96d4d51170931d5d, State: Running, Role: LEADER
I20260620 03:04:16.619117 21836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b48823b21864f3e96d4d51170931d5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33157 } }
I20260620 03:04:16.619292 21826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:16.627298 21838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b48823b21864f3e96d4d51170931d5d. Latest consensus state: current_term: 1 leader_uuid: "2b48823b21864f3e96d4d51170931d5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b48823b21864f3e96d4d51170931d5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33157 } } }
I20260620 03:04:16.628729 21838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:16.629534 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:16.612688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:50215 (local address 127.0.0.1:33157)
0620 03:04:16.612906 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:16.612914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:16.613451 (+ 537us) server_negotiation.cc:407] Connection header received
0620 03:04:16.613674 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:16.613678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:16.613749 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:16.613828 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:16.615705 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:16.616514 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:16.618236 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:16.618945 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:16.619084 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:16.621935 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:16.621941 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:16.621946 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:16.621984 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:16.624813 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:16.625634 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:16.625639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:16.625642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:16.625849 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:16.626445 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:16.626450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:16.626453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:16.626907 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:16.627218 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:16.628334 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0620 03:04:16.628560 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:04:16.628743 21837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b48823b21864f3e96d4d51170931d5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b48823b21864f3e96d4d51170931d5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33157 } } }
I20260620 03:04:16.630344 21837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:16.631119 21846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:04:16.636371 21846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:04:16.645855 21846 catalog_manager.cc:1347] Generated new cluster ID: 7dfd6d284556420986e2710f5b4d8627
I20260620 03:04:16.646008 21846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:04:16.655292 21846 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:04:16.656800 21846 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:04:16.667857 21846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b48823b21864f3e96d4d51170931d5d: Generated new TSK 0
I20260620 03:04:16.668721 21846 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:04:16.674304 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 21759
I20260620 03:04:16.676268 21821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:04:16.765702 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 21759
W20260620 03:04:16.779942 21744 connection.cc:535] client connection to 127.0.0.1:33157 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33157): Connection reset by peer (error 104)
2026-06-20T03:04:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1986 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:04:16 dist-test-slave-f1j0 krb5kdc[21858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:04:16 dist-test-slave-f1j0 krb5kdc[21858](info): set up 2 sockets
Jun 20 03:04:16 dist-test-slave-f1j0 krb5kdc[21858](info): commencing operation
krb5kdc: starting...
W20260620 03:04:17.877982 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:04:18 dist-test-slave-f1j0 krb5kdc[21858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:04:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:04:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:04:18.132335 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43435
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:04:18.462905 21871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:18.463270 21871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:18.463465 21871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:18.470836 21871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:18.470968 21871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:18.471068 21871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:18.471181 21871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:18.471259 21871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:18.481467 21871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43435
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:18.482857 21871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:18.484236 21871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:18.496446 21876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:18.496434 21877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:18.496814 21879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:18.497164 21871 server_base.cc:1034] running on GCE node
I20260620 03:04:18.498107 21871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:18.499598 21871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:18.500803 21871 hybrid_clock.cc:648] HybridClock initialized: now 1781924658500759 us; error 32 us; skew 500 ppm
I20260620 03:04:18.501156 21871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:18 dist-test-slave-f1j0 krb5kdc[21858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:18.504341 21871 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:04:18.506075 21871 webserver.cc:460] Webserver started at http://127.0.0.1:40719/ using document root <none> and password file <none>
I20260620 03:04:18.506804 21871 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:18.506937 21871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:18.507261 21871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:18.510375 21871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "058ed853214c4b67972d75fc3032fed6"
format_stamp: "Formatted at 2026-06-20 03:04:18 on dist-test-slave-f1j0"
server_key: "7203e77953602b4abcefdc4b60a46d12"
server_key_iv: "99624b6c03d51fdd4a06ebf5652622f2"
server_key_version: "encryptionkey@0"
I20260620 03:04:18.511155 21871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "058ed853214c4b67972d75fc3032fed6"
format_stamp: "Formatted at 2026-06-20 03:04:18 on dist-test-slave-f1j0"
server_key: "7203e77953602b4abcefdc4b60a46d12"
server_key_iv: "99624b6c03d51fdd4a06ebf5652622f2"
server_key_version: "encryptionkey@0"
I20260620 03:04:18.516402 21871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:04:18.519968 21886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:18.521303 21871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 03:04:18.521487 21871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "058ed853214c4b67972d75fc3032fed6"
format_stamp: "Formatted at 2026-06-20 03:04:18 on dist-test-slave-f1j0"
server_key: "7203e77953602b4abcefdc4b60a46d12"
server_key_iv: "99624b6c03d51fdd4a06ebf5652622f2"
server_key_version: "encryptionkey@0"
I20260620 03:04:18.521742 21871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:18.536636 21871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:18.540695 21871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:18.541000 21871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:18.559468 21871 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43435
I20260620 03:04:18.559499 21937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43435 every 8 connection(s)
I20260620 03:04:18.560839 21871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:04:18.564658 21938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:18.567963 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 21871
I20260620 03:04:18.568146 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:04:18.568681 388 external_mini_cluster.cc:1428] Setting key 5829cd53794a016096c5f6614a8e4738
I20260620 03:04:18.577368 21938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6: Bootstrap starting.
I20260620 03:04:18.581053 21938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:18.582260 21938 log.cc:824] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:18.582659 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:18.570541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:32831 (local address 127.0.0.1:43435)
0620 03:04:18.571314 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:18.571325 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:18.571366 (+ 41us) server_negotiation.cc:407] Connection header received
0620 03:04:18.573061 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:18.573084 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:18.573515 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:18.574264 (+ 749us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:18.574764 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:18.576086 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:18.577959 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:18.578665 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:18.578817 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:18.580191 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:18.580198 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:18.580212 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:18.580762 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:18.580845 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:18.580869 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:18.581014 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:18.581395 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 03:04:18.581627 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":355,"threads_started":1}
I20260620 03:04:18.585075 21938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6: No bootstrap required, opened a new log
I20260620 03:04:18.586321 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:04:18.595728 21938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "058ed853214c4b67972d75fc3032fed6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43435 } }
I20260620 03:04:18.596237 21938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:04:18.596375 21938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058ed853214c4b67972d75fc3032fed6, State: Initialized, Role: FOLLOWER
I20260620 03:04:18.597023 21938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "058ed853214c4b67972d75fc3032fed6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43435 } }
I20260620 03:04:18.597314 21938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:04:18.597460 21938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:04:18.597676 21938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:18.598613 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:18.591611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:46687 (local address 127.0.0.1:43435)
0620 03:04:18.591821 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:18.591828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:18.592146 (+ 318us) server_negotiation.cc:407] Connection header received
0620 03:04:18.592340 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:18.592344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:18.592408 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:18.592487 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:18.593123 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:18.593999 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:18.595920 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:18.596466 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:18.596590 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:18.597845 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:18.597850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:18.597853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:18.597870 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:18.597906 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:18.597920 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:18.598050 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:18.598250 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 03:04:18.598306 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 03:04:18.600775 21938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "058ed853214c4b67972d75fc3032fed6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43435 } }
I20260620 03:04:18.601280 21938 leader_election.cc:302] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 058ed853214c4b67972d75fc3032fed6; no voters:
I20260620 03:04:18.601886 21938 leader_election.cc:288] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:04:18.602020 21948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:04:18.602707 21948 raft_consensus.cc:686] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6 [term 1 LEADER]: Becoming Leader. State: Replica: 058ed853214c4b67972d75fc3032fed6, State: Running, Role: LEADER
I20260620 03:04:18.603269 21948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "058ed853214c4b67972d75fc3032fed6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43435 } }
I20260620 03:04:18.604020 21938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:18.605849 21949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "058ed853214c4b67972d75fc3032fed6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "058ed853214c4b67972d75fc3032fed6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43435 } } }
I20260620 03:04:18.606232 21949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:18.608279 21950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 058ed853214c4b67972d75fc3032fed6. Latest consensus state: current_term: 1 leader_uuid: "058ed853214c4b67972d75fc3032fed6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "058ed853214c4b67972d75fc3032fed6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43435 } } }
I20260620 03:04:18.608659 21950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:18.610033 21954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:04:18.615662 21954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:04:18.624363 21954 catalog_manager.cc:1347] Generated new cluster ID: bfbf881e10bd43d79c828913d05f4823
I20260620 03:04:18.624485 21954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:04:18.636737 21954 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:04:18.638260 21954 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:04:18.647238 21954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 058ed853214c4b67972d75fc3032fed6: Generated new TSK 0
I20260620 03:04:18.648098 21954 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:04:18.661087 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 21871
W20260620 03:04:18.662439 21933 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.99:32831
W20260620 03:04:18.663194 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 21871: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:04:18.663348 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 21871
W20260620 03:04:18.680797 21856 connection.cc:535] client connection to 127.0.0.1:43435 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43435): Connection reset by peer (error 104)
2026-06-20T03:04:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1900 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:04:18 dist-test-slave-f1j0 krb5kdc[21969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:04:18 dist-test-slave-f1j0 krb5kdc[21969](info): set up 2 sockets
Jun 20 03:04:18 dist-test-slave-f1j0 krb5kdc[21969](info): commencing operation
krb5kdc: starting...
W20260620 03:04:19.762190 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:04:19 dist-test-slave-f1j0 krb5kdc[21969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:04:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:04:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 03:04:20.025182 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32903
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:04:20.364773 21982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:20.365063 21982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:20.365242 21982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:20.372604 21982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:20.372714 21982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:20.372844 21982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:20.372931 21982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:20.373023 21982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:20.383253 21982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32903
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:20.384585 21982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:20.385942 21982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:20.397842 21990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:20.397903 21987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:20.397955 21988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:20.398604 21982 server_base.cc:1034] running on GCE node
I20260620 03:04:20.399411 21982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:20.400843 21982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:20.402113 21982 hybrid_clock.cc:648] HybridClock initialized: now 1781924660402068 us; error 70 us; skew 500 ppm
I20260620 03:04:20.402515 21982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:20 dist-test-slave-f1j0 krb5kdc[21969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:20.410749 21982 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 03:04:20.412410 21982 webserver.cc:460] Webserver started at http://127.0.0.1:36119/ using document root <none> and password file <none>
I20260620 03:04:20.413169 21982 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:20.413321 21982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:20.413727 21982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:20.416754 21982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "9a28f7534f6044f0b1ab226cac5f4fb9"
format_stamp: "Formatted at 2026-06-20 03:04:20 on dist-test-slave-f1j0"
server_key: "dc58f3b6de20c9ed65cf27446b0a2856"
server_key_iv: "6cb4e33bf9bdeeb4422242a0c5d72a5c"
server_key_version: "encryptionkey@0"
I20260620 03:04:20.417521 21982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "9a28f7534f6044f0b1ab226cac5f4fb9"
format_stamp: "Formatted at 2026-06-20 03:04:20 on dist-test-slave-f1j0"
server_key: "dc58f3b6de20c9ed65cf27446b0a2856"
server_key_iv: "6cb4e33bf9bdeeb4422242a0c5d72a5c"
server_key_version: "encryptionkey@0"
I20260620 03:04:20.422788 21982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 03:04:20.426451 21997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:20.427811 21982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 03:04:20.427991 21982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "9a28f7534f6044f0b1ab226cac5f4fb9"
format_stamp: "Formatted at 2026-06-20 03:04:20 on dist-test-slave-f1j0"
server_key: "dc58f3b6de20c9ed65cf27446b0a2856"
server_key_iv: "6cb4e33bf9bdeeb4422242a0c5d72a5c"
server_key_version: "encryptionkey@0"
I20260620 03:04:20.428198 21982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:20.434172 21982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:20.438699 21982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:20.439005 21982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:20.458089 21982 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32903
I20260620 03:04:20.458135 22048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32903 every 8 connection(s)
I20260620 03:04:20.459570 21982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:04:20.460983 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 21982
I20260620 03:04:20.461169 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:04:20.461597 388 external_mini_cluster.cc:1428] Setting key f672d99cf40ae3c74fe50d6e4120027c
I20260620 03:04:20.464432 22049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:20.471328 22052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:20.463240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:59847 (local address 127.0.0.1:32903)
0620 03:04:20.464025 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:20.464037 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:20.466122 (+ 2085us) server_negotiation.cc:407] Connection header received
0620 03:04:20.467826 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:20.467845 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:20.468189 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:20.468608 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:20.469702 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:20.469708 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:20.469720 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:20.469734 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:20.469804 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:20.469826 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:20.470084 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:20.470238 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 03:04:20.470421 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":368,"threads_started":1}
I20260620 03:04:20.474895 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:04:20.476819 22049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9: Bootstrap starting.
I20260620 03:04:20.480393 22049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:20.481748 22049 log.cc:824] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:20.481950 22052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:20.479391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:51527 (local address 127.0.0.1:32903)
0620 03:04:20.479588 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:20.479594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:20.479948 (+ 354us) server_negotiation.cc:407] Connection header received
0620 03:04:20.480113 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:20.480117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:20.480161 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:20.480238 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:20.481176 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:20.481180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:20.481183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:20.481199 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:20.481236 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:20.481250 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:20.481339 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:20.481495 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 03:04:20.481546 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 03:04:20.484731 22049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9: No bootstrap required, opened a new log
I20260620 03:04:20.492322 22049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a28f7534f6044f0b1ab226cac5f4fb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } }
I20260620 03:04:20.492719 22049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:04:20.492838 22049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a28f7534f6044f0b1ab226cac5f4fb9, State: Initialized, Role: FOLLOWER
I20260620 03:04:20.493475 22049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a28f7534f6044f0b1ab226cac5f4fb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } }
I20260620 03:04:20.493772 22049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:04:20.493927 22049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:04:20.494097 22049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:20.496543 22049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a28f7534f6044f0b1ab226cac5f4fb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } }
I20260620 03:04:20.497074 22049 leader_election.cc:302] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a28f7534f6044f0b1ab226cac5f4fb9; no voters:
I20260620 03:04:20.497692 22049 leader_election.cc:288] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:04:20.497828 22059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:04:20.498510 22059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9 [term 1 LEADER]: Becoming Leader. State: Replica: 9a28f7534f6044f0b1ab226cac5f4fb9, State: Running, Role: LEADER
I20260620 03:04:20.499071 22059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a28f7534f6044f0b1ab226cac5f4fb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } }
I20260620 03:04:20.499871 22049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:20.501876 22061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a28f7534f6044f0b1ab226cac5f4fb9. Latest consensus state: current_term: 1 leader_uuid: "9a28f7534f6044f0b1ab226cac5f4fb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a28f7534f6044f0b1ab226cac5f4fb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } } }
I20260620 03:04:20.502338 22061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:20.501847 22060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a28f7534f6044f0b1ab226cac5f4fb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a28f7534f6044f0b1ab226cac5f4fb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } } }
I20260620 03:04:20.502442 22060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:20.504277 22065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:04:20.509442 22065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:04:20.519378 22065 catalog_manager.cc:1347] Generated new cluster ID: c8381f354b1c4727917e94fececdc5bd
I20260620 03:04:20.519538 22065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:04:20.530715 22065 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:04:20.531178 22065 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:04:20.543746 22065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a28f7534f6044f0b1ab226cac5f4fb9: Generated new TSK 0
I20260620 03:04:20.544564 22065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:04:20.553640 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 21982
W20260620 03:04:20.554735 22044 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.99:59847
W20260620 03:04:20.555466 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 21982: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:04:20.555619 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 21982
2026-06-20T03:04:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1891 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:04:20 dist-test-slave-f1j0 krb5kdc[22080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:04:20 dist-test-slave-f1j0 krb5kdc[22080](info): set up 2 sockets
Jun 20 03:04:20 dist-test-slave-f1j0 krb5kdc[22080](info): commencing operation
krb5kdc: starting...
W20260620 03:04:21.657984 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:04:21 dist-test-slave-f1j0 krb5kdc[22080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:04:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:04:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:04:21.912940 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35921
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:04:22.243985 22093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:22.244356 22093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:22.244534 22093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:22.251820 22093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:22.251988 22093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:22.252075 22093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:22.252169 22093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:22.252234 22093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:22.261996 22093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35921
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:22.263352 22093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:22.264720 22093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:22.276815 22098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:22.276744 22101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:22.276790 22099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:22.277379 22093 server_base.cc:1034] running on GCE node
I20260620 03:04:22.278304 22093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:22.279781 22093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:22.281023 22093 hybrid_clock.cc:648] HybridClock initialized: now 1781924662280975 us; error 41 us; skew 500 ppm
I20260620 03:04:22.281383 22093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:22 dist-test-slave-f1j0 krb5kdc[22080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:22.284749 22093 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:04:22.286311 22093 webserver.cc:460] Webserver started at http://127.0.97.62:35139/ using document root <none> and password file <none>
I20260620 03:04:22.287027 22093 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:22.287190 22093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:22.287511 22093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:22.290589 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "6fd07148ea214f5a8547061f71030a21"
format_stamp: "Formatted at 2026-06-20 03:04:22 on dist-test-slave-f1j0"
server_key: "e34d03015c047c1dfd923e2daf7f8f07"
server_key_iv: "cb899c6778f4737953b30fa2de658249"
server_key_version: "encryptionkey@0"
I20260620 03:04:22.291348 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "6fd07148ea214f5a8547061f71030a21"
format_stamp: "Formatted at 2026-06-20 03:04:22 on dist-test-slave-f1j0"
server_key: "e34d03015c047c1dfd923e2daf7f8f07"
server_key_iv: "cb899c6778f4737953b30fa2de658249"
server_key_version: "encryptionkey@0"
I20260620 03:04:22.296497 22093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 03:04:22.299940 22108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:22.301250 22093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 03:04:22.301429 22093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "6fd07148ea214f5a8547061f71030a21"
format_stamp: "Formatted at 2026-06-20 03:04:22 on dist-test-slave-f1j0"
server_key: "e34d03015c047c1dfd923e2daf7f8f07"
server_key_iv: "cb899c6778f4737953b30fa2de658249"
server_key_version: "encryptionkey@0"
I20260620 03:04:22.301702 22093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:22.325342 22093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:22.329471 22093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:22.329826 22093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:22.348704 22093 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35921
I20260620 03:04:22.348753 22159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35921 every 8 connection(s)
I20260620 03:04:22.350214 22093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:04:22.354079 22160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:22.357456 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 22093
I20260620 03:04:22.357669 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:04:22.358204 388 external_mini_cluster.cc:1428] Setting key c967292b762e5637d7b814078555a52d
I20260620 03:04:22.365957 22160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21: Bootstrap starting.
I20260620 03:04:22.368988 22160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:22.370133 22160 log.cc:824] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21: Log is configured to *not* fsync() on all Append() calls
Jun 20 03:04:22 dist-test-slave-f1j0 krb5kdc[22080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:04:22.373181 22160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21: No bootstrap required, opened a new log
I20260620 03:04:22.381611 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:22.360072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60793 (local address 127.0.97.62:35921)
0620 03:04:22.360822 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:22.360835 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:22.360877 (+ 42us) server_negotiation.cc:407] Connection header received
0620 03:04:22.362562 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:22.362597 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:22.363015 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:22.363415 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:22.364838 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:22.366264 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:22.367541 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:22.368206 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:22.368411 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:22.372732 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:22.372743 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:22.372761 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:22.372809 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:22.376835 (+ 4026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:22.377755 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:22.377768 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:22.377777 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:22.377987 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:22.378427 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:22.378432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:22.378436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:22.379129 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:22.379463 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:22.379884 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 03:04:22.380223 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":326,"threads_started":1}
I20260620 03:04:22.384109 22160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd07148ea214f5a8547061f71030a21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35921 } }
I20260620 03:04:22.384581 22160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:04:22.384704 22160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd07148ea214f5a8547061f71030a21, State: Initialized, Role: FOLLOWER
I20260620 03:04:22.385358 22160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd07148ea214f5a8547061f71030a21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35921 } }
I20260620 03:04:22.385653 22160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:04:22.385795 22160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:04:22.385869 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:04:22.385964 22160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:22.389158 22160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd07148ea214f5a8547061f71030a21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35921 } }
I20260620 03:04:22.389864 22160 leader_election.cc:302] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fd07148ea214f5a8547061f71030a21; no voters:
I20260620 03:04:22.390548 22160 leader_election.cc:288] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:04:22.390617 22169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:04:22.392454 22169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21 [term 1 LEADER]: Becoming Leader. State: Replica: 6fd07148ea214f5a8547061f71030a21, State: Running, Role: LEADER
I20260620 03:04:22.393193 22169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd07148ea214f5a8547061f71030a21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35921 } }
I20260620 03:04:22.393971 22160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:22.395886 22170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fd07148ea214f5a8547061f71030a21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd07148ea214f5a8547061f71030a21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35921 } } }
I20260620 03:04:22.396270 22170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:22.396785 22171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fd07148ea214f5a8547061f71030a21. Latest consensus state: current_term: 1 leader_uuid: "6fd07148ea214f5a8547061f71030a21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd07148ea214f5a8547061f71030a21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35921 } } }
I20260620 03:04:22.397089 22171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:22.401688 22175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:04:22.412822 22175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:04:22.427464 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:22.391279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48491 (local address 127.0.97.62:35921)
0620 03:04:22.391508 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:22.391517 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:22.404400 (+ 12883us) server_negotiation.cc:407] Connection header received
0620 03:04:22.405084 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:22.405093 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:22.405202 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:22.405292 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:22.408835 (+ 3543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:22.410498 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:22.413045 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:22.414027 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:22.414161 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:22.418256 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:22.418266 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:22.418274 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:22.418328 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:22.423078 (+ 4750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:22.424314 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:22.424323 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:22.424329 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:22.424528 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:22.425235 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:22.425243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:22.425249 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:22.425777 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:22.426089 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:22.426891 (+ 802us) server_negotiation.cc:299] Negotiation successful
0620 03:04:22.427092 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 03:04:22.429704 22175 catalog_manager.cc:1347] Generated new cluster ID: f32f5d65fba74c9b81496a5648f52203
I20260620 03:04:22.429839 22175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:04:22.448273 22175 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:04:22.450871 22175 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:04:22.469247 22175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fd07148ea214f5a8547061f71030a21: Generated new TSK 0
I20260620 03:04:22.470391 22175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:04:22.480993 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 22093
I20260620 03:04:22.482457 22155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:04:22.595172 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 22093
W20260620 03:04:22.612756 22078 connection.cc:535] client connection to 127.0.97.62:35921 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:35921): Connection reset by peer (error 104)
2026-06-20T03:04:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2039 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:04:22 dist-test-slave-f1j0 krb5kdc[22193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:04:22 dist-test-slave-f1j0 krb5kdc[22193](info): set up 2 sockets
Jun 20 03:04:22 dist-test-slave-f1j0 krb5kdc[22193](info): commencing operation
krb5kdc: starting...
W20260620 03:04:23.690667 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:04:23 dist-test-slave-f1j0 krb5kdc[22193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:04:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:04:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:04:23.944875 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42729
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:04:24.278262 22206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:24.278545 22206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:24.278636 22206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:24.285612 22206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:24.285705 22206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:24.285751 22206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:24.285810 22206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:24.285848 22206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:24.295392 22206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42729
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:24.296622 22206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:24.297891 22206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:24.309680 22214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:24.309729 22212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:24.309774 22211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:24.315128 22206 server_base.cc:1034] running on GCE node
I20260620 03:04:24.315868 22206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:24.317217 22206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:24.318415 22206 hybrid_clock.cc:648] HybridClock initialized: now 1781924664318373 us; error 31 us; skew 500 ppm
I20260620 03:04:24.318792 22206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:24 dist-test-slave-f1j0 krb5kdc[22193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:24.321933 22206 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:04:24.323386 22206 webserver.cc:460] Webserver started at http://127.0.97.62:38245/ using document root <none> and password file <none>
I20260620 03:04:24.324105 22206 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:24.324247 22206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:24.324570 22206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:24.327620 22206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "02bee0c2f7464f98ab4895e5d5e508e4"
format_stamp: "Formatted at 2026-06-20 03:04:24 on dist-test-slave-f1j0"
server_key: "b9b5338c9b9b8baacac75a17c748f10a"
server_key_iv: "92cea1e8c38d1b4aa9137057081cb92d"
server_key_version: "encryptionkey@0"
I20260620 03:04:24.328415 22206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "02bee0c2f7464f98ab4895e5d5e508e4"
format_stamp: "Formatted at 2026-06-20 03:04:24 on dist-test-slave-f1j0"
server_key: "b9b5338c9b9b8baacac75a17c748f10a"
server_key_iv: "92cea1e8c38d1b4aa9137057081cb92d"
server_key_version: "encryptionkey@0"
I20260620 03:04:24.333652 22206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 03:04:24.336999 22221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:24.338402 22206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 03:04:24.338590 22206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "02bee0c2f7464f98ab4895e5d5e508e4"
format_stamp: "Formatted at 2026-06-20 03:04:24 on dist-test-slave-f1j0"
server_key: "b9b5338c9b9b8baacac75a17c748f10a"
server_key_iv: "92cea1e8c38d1b4aa9137057081cb92d"
server_key_version: "encryptionkey@0"
I20260620 03:04:24.338804 22206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:24.354154 22206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:24.358314 22206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:24.358599 22206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:24.377431 22206 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42729
I20260620 03:04:24.377465 22272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42729 every 8 connection(s)
I20260620 03:04:24.378880 22206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:04:24.382681 22273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:24.388883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 22206
I20260620 03:04:24.389084 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:04:24.389667 388 external_mini_cluster.cc:1428] Setting key 939f19a6b1b1a180e0ed703ded62db20
I20260620 03:04:24.394088 22273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4: Bootstrap starting.
I20260620 03:04:24.397161 22273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:24.398238 22273 log.cc:824] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:24.401511 22273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4: No bootstrap required, opened a new log
I20260620 03:04:24.402902 22276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:24.391452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34385 (local address 127.0.97.62:42729)
0620 03:04:24.392251 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:24.392263 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:24.392303 (+ 40us) server_negotiation.cc:407] Connection header received
0620 03:04:24.393995 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:24.394018 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:24.394464 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:24.394864 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:24.395411 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:24.396698 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:24.398160 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:24.399218 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:24.399432 (+ 214us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:24.400649 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:24.400657 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:24.400671 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:24.401174 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:24.401246 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:24.401271 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:24.401384 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:24.401795 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 03:04:24.401986 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":367,"threads_started":1}
I20260620 03:04:24.406770 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:04:24.410948 22273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02bee0c2f7464f98ab4895e5d5e508e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42729 } }
I20260620 03:04:24.411408 22273 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:04:24.411514 22273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02bee0c2f7464f98ab4895e5d5e508e4, State: Initialized, Role: FOLLOWER
I20260620 03:04:24.412271 22273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02bee0c2f7464f98ab4895e5d5e508e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42729 } }
I20260620 03:04:24.412533 22273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:04:24.412684 22273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:04:24.412885 22273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:24.415931 22273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02bee0c2f7464f98ab4895e5d5e508e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42729 } }
I20260620 03:04:24.416527 22273 leader_election.cc:302] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02bee0c2f7464f98ab4895e5d5e508e4; no voters:
I20260620 03:04:24.417254 22273 leader_election.cc:288] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:04:24.417382 22283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:04:24.418874 22283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4 [term 1 LEADER]: Becoming Leader. State: Replica: 02bee0c2f7464f98ab4895e5d5e508e4, State: Running, Role: LEADER
I20260620 03:04:24.419513 22283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02bee0c2f7464f98ab4895e5d5e508e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42729 } }
I20260620 03:04:24.419793 22276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:24.412017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41133 (local address 127.0.97.62:42729)
0620 03:04:24.412217 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:24.412224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:24.412787 (+ 563us) server_negotiation.cc:407] Connection header received
0620 03:04:24.413001 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:24.413006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:24.413104 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:24.413218 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:24.413756 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:24.414943 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:24.416265 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:24.416955 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:24.417127 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:24.418555 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:24.418568 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:24.418574 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:24.418595 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:24.418670 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:24.418696 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:24.418812 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:24.419293 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 03:04:24.419347 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:04:24.419762 22273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:24.422879 22284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02bee0c2f7464f98ab4895e5d5e508e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02bee0c2f7464f98ab4895e5d5e508e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42729 } } }
I20260620 03:04:24.423259 22284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:24.422983 22285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02bee0c2f7464f98ab4895e5d5e508e4. Latest consensus state: current_term: 1 leader_uuid: "02bee0c2f7464f98ab4895e5d5e508e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02bee0c2f7464f98ab4895e5d5e508e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42729 } } }
I20260620 03:04:24.423378 22285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:24.426345 22289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:04:24.431941 22289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:04:24.443207 22289 catalog_manager.cc:1347] Generated new cluster ID: 446469eed1954538ae39e8faa1c6d7df
I20260620 03:04:24.443379 22289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:04:24.454759 22289 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:04:24.456065 22289 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:04:24.464208 22289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02bee0c2f7464f98ab4895e5d5e508e4: Generated new TSK 0
I20260620 03:04:24.465129 22289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:04:24.474895 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 22206
W20260620 03:04:24.476759 22268 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34385
W20260620 03:04:24.477566 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 22206: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:04:24.477767 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 22206
W20260620 03:04:24.491889 22191 connection.cc:535] client connection to 127.0.97.62:42729 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:42729): Connection reset by peer (error 104)
2026-06-20T03:04:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1879 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:04:24 dist-test-slave-f1j0 krb5kdc[22304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:04:24 dist-test-slave-f1j0 krb5kdc[22304](info): set up 2 sockets
Jun 20 03:04:24 dist-test-slave-f1j0 krb5kdc[22304](info): commencing operation
krb5kdc: starting...
W20260620 03:04:25.582065 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:04:25 dist-test-slave-f1j0 krb5kdc[22304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:04:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:04:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:04:25.838634 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35013
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:04:26.172376 22317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:26.172684 22317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:26.172781 22317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:26.180052 22317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:26.180178 22317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:26.180233 22317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:26.180296 22317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:26.180341 22317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:26.190338 22317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35013
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:26.191643 22317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:26.192905 22317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:26.205202 22322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:26.205178 22323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:26.205325 22325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:26.205861 22317 server_base.cc:1034] running on GCE node
I20260620 03:04:26.206651 22317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:26.208050 22317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:26.209281 22317 hybrid_clock.cc:648] HybridClock initialized: now 1781924666209240 us; error 72 us; skew 500 ppm
I20260620 03:04:26.209786 22317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:26 dist-test-slave-f1j0 krb5kdc[22304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:26.217661 22317 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:04:26.219165 22317 webserver.cc:460] Webserver started at http://127.0.97.62:39823/ using document root <none> and password file <none>
I20260620 03:04:26.219887 22317 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:26.220046 22317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:26.220358 22317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:26.223526 22317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "4bf30e2e26844ae7affd108c27ab2fa2"
format_stamp: "Formatted at 2026-06-20 03:04:26 on dist-test-slave-f1j0"
server_key: "4efeb0b1234b54ee5756ed0d1f43b4f6"
server_key_iv: "959a1595b06901038565bac841005554"
server_key_version: "encryptionkey@0"
I20260620 03:04:26.224319 22317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "4bf30e2e26844ae7affd108c27ab2fa2"
format_stamp: "Formatted at 2026-06-20 03:04:26 on dist-test-slave-f1j0"
server_key: "4efeb0b1234b54ee5756ed0d1f43b4f6"
server_key_iv: "959a1595b06901038565bac841005554"
server_key_version: "encryptionkey@0"
I20260620 03:04:26.229796 22317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 03:04:26.233384 22332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:26.235092 22317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:04:26.235302 22317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "4bf30e2e26844ae7affd108c27ab2fa2"
format_stamp: "Formatted at 2026-06-20 03:04:26 on dist-test-slave-f1j0"
server_key: "4efeb0b1234b54ee5756ed0d1f43b4f6"
server_key_iv: "959a1595b06901038565bac841005554"
server_key_version: "encryptionkey@0"
I20260620 03:04:26.235582 22317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:26.242381 22317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:26.246488 22317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:26.246788 22317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:26.265496 22317 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35013
I20260620 03:04:26.265534 22383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35013 every 8 connection(s)
I20260620 03:04:26.266970 22317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:04:26.270787 22384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:26.273134 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 22317
I20260620 03:04:26.273311 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:04:26.273881 388 external_mini_cluster.cc:1428] Setting key 64d49a9b09617ec47d7cc72735699edc
I20260620 03:04:26.282063 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:26.275762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44453 (local address 127.0.97.62:35013)
0620 03:04:26.276652 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:26.276664 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:26.276709 (+ 45us) server_negotiation.cc:407] Connection header received
0620 03:04:26.278464 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:26.278487 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:26.278858 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:26.279196 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:26.280292 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:26.280299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:26.280312 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:26.280328 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:26.280398 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:26.280425 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:26.280656 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:26.280759 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 03:04:26.281018 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":412,"threads_started":1}
I20260620 03:04:26.283273 22384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2: Bootstrap starting.
I20260620 03:04:26.286062 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:04:26.286993 22384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:26.288239 22384 log.cc:824] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:26.291193 22384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2: No bootstrap required, opened a new log
I20260620 03:04:26.293901 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:26.291218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34369 (local address 127.0.97.62:35013)
0620 03:04:26.291405 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:26.291412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:26.291928 (+ 516us) server_negotiation.cc:407] Connection header received
0620 03:04:26.292056 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:26.292061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:26.292118 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:26.292227 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:26.293131 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:26.293163 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:26.293169 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:26.293188 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:26.293247 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:26.293266 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:26.293378 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:26.293528 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 03:04:26.293617 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 03:04:26.299242 22384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf30e2e26844ae7affd108c27ab2fa2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35013 } }
I20260620 03:04:26.299597 22384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:04:26.299669 22384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf30e2e26844ae7affd108c27ab2fa2, State: Initialized, Role: FOLLOWER
I20260620 03:04:26.300235 22384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf30e2e26844ae7affd108c27ab2fa2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35013 } }
I20260620 03:04:26.300431 22384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:04:26.300527 22384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:04:26.300643 22384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:26.302770 22384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf30e2e26844ae7affd108c27ab2fa2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35013 } }
I20260620 03:04:26.303212 22384 leader_election.cc:302] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bf30e2e26844ae7affd108c27ab2fa2; no voters:
I20260620 03:04:26.303730 22384 leader_election.cc:288] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:04:26.303927 22394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:04:26.304662 22394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2 [term 1 LEADER]: Becoming Leader. State: Replica: 4bf30e2e26844ae7affd108c27ab2fa2, State: Running, Role: LEADER
I20260620 03:04:26.305311 22394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf30e2e26844ae7affd108c27ab2fa2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35013 } }
I20260620 03:04:26.306375 22384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:26.308210 22396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bf30e2e26844ae7affd108c27ab2fa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf30e2e26844ae7affd108c27ab2fa2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35013 } } }
I20260620 03:04:26.308202 22395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bf30e2e26844ae7affd108c27ab2fa2. Latest consensus state: current_term: 1 leader_uuid: "4bf30e2e26844ae7affd108c27ab2fa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf30e2e26844ae7affd108c27ab2fa2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35013 } } }
I20260620 03:04:26.308779 22395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:26.308779 22396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:26.310357 22400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:04:26.315910 22400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:04:26.324762 22400 catalog_manager.cc:1347] Generated new cluster ID: d907ccb0f353489aa65c257f6b5bf1a8
I20260620 03:04:26.324900 22400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:04:26.351339 22400 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:04:26.351778 22400 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:04:26.363552 22400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bf30e2e26844ae7affd108c27ab2fa2: Generated new TSK 0
I20260620 03:04:26.364441 22400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:04:26.377662 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 22317
W20260620 03:04:26.379014 22379 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44453
W20260620 03:04:26.379801 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 22317: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:04:26.379961 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 22317
2026-06-20T03:04:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1901 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:04:26 dist-test-slave-f1j0 krb5kdc[22415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:04:26 dist-test-slave-f1j0 krb5kdc[22415](info): set up 2 sockets
Jun 20 03:04:26 dist-test-slave-f1j0 krb5kdc[22415](info): commencing operation
krb5kdc: starting...
W20260620 03:04:27.486145 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:04:27 dist-test-slave-f1j0 krb5kdc[22415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:04:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:04:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:04:27.750880 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40729
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:04:28.078887 22428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:28.079278 22428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:28.079485 22428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:28.087255 22428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:28.087420 22428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:28.087532 22428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:28.087642 22428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:28.087726 22428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:28.098047 22428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:28.099488 22428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:28.100829 22428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:28.112962 22434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:28.113149 22436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:28.113446 22433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:28.114521 22428 server_base.cc:1034] running on GCE node
I20260620 03:04:28.115574 22428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:28.117808 22428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:28.119051 22428 hybrid_clock.cc:648] HybridClock initialized: now 1781924668119004 us; error 31 us; skew 500 ppm
I20260620 03:04:28.119426 22428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:28 dist-test-slave-f1j0 krb5kdc[22415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:28.123224 22428 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:04:28.125051 22428 webserver.cc:460] Webserver started at http://127.0.97.62:35189/ using document root <none> and password file <none>
I20260620 03:04:28.125829 22428 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:28.125977 22428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:28.126291 22428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:28.129304 22428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "493bdae1b6c344fca7f236b63dabcce6"
format_stamp: "Formatted at 2026-06-20 03:04:28 on dist-test-slave-f1j0"
server_key: "22a374cf1e6359d18078a88574930548"
server_key_iv: "e146ea935c4c6169ba4fbb2663f16a67"
server_key_version: "encryptionkey@0"
I20260620 03:04:28.130142 22428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "493bdae1b6c344fca7f236b63dabcce6"
format_stamp: "Formatted at 2026-06-20 03:04:28 on dist-test-slave-f1j0"
server_key: "22a374cf1e6359d18078a88574930548"
server_key_iv: "e146ea935c4c6169ba4fbb2663f16a67"
server_key_version: "encryptionkey@0"
I20260620 03:04:28.135237 22428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:04:28.138780 22443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:28.140133 22428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 03:04:28.140340 22428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "493bdae1b6c344fca7f236b63dabcce6"
format_stamp: "Formatted at 2026-06-20 03:04:28 on dist-test-slave-f1j0"
server_key: "22a374cf1e6359d18078a88574930548"
server_key_iv: "e146ea935c4c6169ba4fbb2663f16a67"
server_key_version: "encryptionkey@0"
I20260620 03:04:28.140559 22428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:28.168315 22428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:28.172941 22428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:28.173252 22428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:28.192641 22428 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40729
I20260620 03:04:28.192729 22494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40729 every 8 connection(s)
I20260620 03:04:28.194168 22428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:04:28.196177 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 22428
I20260620 03:04:28.196375 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:04:28.196934 388 external_mini_cluster.cc:1428] Setting key 08895ee5344973fbaa5282af5eb92f62
I20260620 03:04:28.198760 22495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:28.209507 22495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6: Bootstrap starting.
Jun 20 03:04:28 dist-test-slave-f1j0 krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:04:28.212620 22495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:28.213732 22495 log.cc:824] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:28.216372 22495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6: No bootstrap required, opened a new log
I20260620 03:04:28.218358 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:28.198842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38993 (local address 127.0.97.62:40729)
0620 03:04:28.199622 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:28.199633 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:28.199674 (+ 41us) server_negotiation.cc:407] Connection header received
0620 03:04:28.201311 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:28.201333 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:28.201820 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:28.202239 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:28.203666 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:28.204955 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:28.206377 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:28.207051 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:28.207272 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:28.211154 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:28.211163 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:28.211176 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:28.211210 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:28.214452 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:28.215105 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:28.215111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:28.215117 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:28.215273 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:28.215666 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:28.215671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:28.215676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:28.216183 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:28.216497 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:28.216938 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 03:04:28.217341 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":328,"threads_started":1}
I20260620 03:04:28.221998 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:04:28.224257 22495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493bdae1b6c344fca7f236b63dabcce6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40729 } }
I20260620 03:04:28.224650 22495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:04:28.224772 22495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493bdae1b6c344fca7f236b63dabcce6, State: Initialized, Role: FOLLOWER
I20260620 03:04:28.225466 22495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493bdae1b6c344fca7f236b63dabcce6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40729 } }
I20260620 03:04:28.225752 22495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:04:28.225904 22495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:04:28.226080 22495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:28.228561 22495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493bdae1b6c344fca7f236b63dabcce6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40729 } }
I20260620 03:04:28.229151 22495 leader_election.cc:302] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 493bdae1b6c344fca7f236b63dabcce6; no voters:
I20260620 03:04:28.229890 22495 leader_election.cc:288] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:04:28.229981 22505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:04:28.231901 22495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:28.237007 22505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6 [term 1 LEADER]: Becoming Leader. State: Replica: 493bdae1b6c344fca7f236b63dabcce6, State: Running, Role: LEADER
I20260620 03:04:28.237872 22505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493bdae1b6c344fca7f236b63dabcce6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40729 } }
I20260620 03:04:28.240684 22506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 493bdae1b6c344fca7f236b63dabcce6. Latest consensus state: current_term: 1 leader_uuid: "493bdae1b6c344fca7f236b63dabcce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493bdae1b6c344fca7f236b63dabcce6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40729 } } }
I20260620 03:04:28.241053 22506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:28.241552 22507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "493bdae1b6c344fca7f236b63dabcce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493bdae1b6c344fca7f236b63dabcce6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40729 } } }
I20260620 03:04:28.241887 22507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:28.246995 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:28.226288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59443 (local address 127.0.97.62:40729)
0620 03:04:28.226469 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:28.226477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:28.227025 (+ 548us) server_negotiation.cc:407] Connection header received
0620 03:04:28.227142 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:28.227147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:28.227235 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:28.227340 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:28.228693 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:28.229883 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:28.232214 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:28.232897 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:28.233046 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:28.236672 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:28.236682 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:28.236691 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:28.236745 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:28.241516 (+ 4771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:28.243254 (+ 1738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:28.243259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:28.243263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:28.243416 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:28.243956 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:28.243961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:28.243964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:28.244385 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:28.245926 (+ 1541us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:28.246117 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 03:04:28.246428 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 03:04:28.257699 22515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:04:28.262531 22515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:04:28.271507 22515 catalog_manager.cc:1347] Generated new cluster ID: 751bf006407b48bfa1d90a0825976cfb
I20260620 03:04:28.271662 22515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:04:28.287969 22515 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:04:28.289362 22515 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:04:28.296052 22515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 493bdae1b6c344fca7f236b63dabcce6: Generated new TSK 0
I20260620 03:04:28.297164 22515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:04:28.313926 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 22428
I20260620 03:04:28.315099 22490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:04:28.409970 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 22428
2026-06-20T03:04:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2033 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:04:28 dist-test-slave-f1j0 krb5kdc[22528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:04:28 dist-test-slave-f1j0 krb5kdc[22528](info): set up 2 sockets
Jun 20 03:04:28 dist-test-slave-f1j0 krb5kdc[22528](info): commencing operation
krb5kdc: starting...
W20260620 03:04:29.526015 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:04:29 dist-test-slave-f1j0 krb5kdc[22528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:04:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:04:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:04:29.782768 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33481
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:04:30.112161 22541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:30.112608 22541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:30.112814 22541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:30.123217 22541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:30.123330 22541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:30.123378 22541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:30.123443 22541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:30.123487 22541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:30.133481 22541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:30.134853 22541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:30.136111 22541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:30.148134 22546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:30.148103 22547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:30.148466 22549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:30.148837 22541 server_base.cc:1034] running on GCE node
I20260620 03:04:30.149721 22541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:30.151081 22541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:30.152334 22541 hybrid_clock.cc:648] HybridClock initialized: now 1781924670152298 us; error 50 us; skew 500 ppm
I20260620 03:04:30.152704 22541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:30 dist-test-slave-f1j0 krb5kdc[22528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:30.155871 22541 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:04:30.157327 22541 webserver.cc:460] Webserver started at http://127.0.97.62:34033/ using document root <none> and password file <none>
I20260620 03:04:30.158073 22541 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:30.158293 22541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:30.158625 22541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:30.161702 22541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "63d4b433991c4bffabb4352abc1df819"
format_stamp: "Formatted at 2026-06-20 03:04:30 on dist-test-slave-f1j0"
server_key: "d8471a0d912b4172bbba0babb0509041"
server_key_iv: "d4f5b525c6f9448ce6c76aab33ceaf32"
server_key_version: "encryptionkey@0"
I20260620 03:04:30.162473 22541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "63d4b433991c4bffabb4352abc1df819"
format_stamp: "Formatted at 2026-06-20 03:04:30 on dist-test-slave-f1j0"
server_key: "d8471a0d912b4172bbba0babb0509041"
server_key_iv: "d4f5b525c6f9448ce6c76aab33ceaf32"
server_key_version: "encryptionkey@0"
I20260620 03:04:30.167569 22541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 03:04:30.171079 22556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:30.172422 22541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 03:04:30.172600 22541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "63d4b433991c4bffabb4352abc1df819"
format_stamp: "Formatted at 2026-06-20 03:04:30 on dist-test-slave-f1j0"
server_key: "d8471a0d912b4172bbba0babb0509041"
server_key_iv: "d4f5b525c6f9448ce6c76aab33ceaf32"
server_key_version: "encryptionkey@0"
I20260620 03:04:30.172792 22541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:30.202342 22541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:30.206689 22541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:30.206977 22541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:30.225032 22541 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33481
I20260620 03:04:30.225080 22607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33481 every 8 connection(s)
I20260620 03:04:30.226356 22541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:04:30.227099 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 22541
I20260620 03:04:30.227294 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:04:30.227814 388 external_mini_cluster.cc:1428] Setting key f26d3027bb016b58919021819a7aba6b
I20260620 03:04:30.230712 22608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:30.240435 22611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:30.229664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60829 (local address 127.0.97.62:33481)
0620 03:04:30.230414 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:30.230426 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:30.230464 (+ 38us) server_negotiation.cc:407] Connection header received
0620 03:04:30.232118 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:30.232139 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:30.232514 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:30.232926 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:30.233469 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:30.234820 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:30.236111 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:30.236765 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:30.236969 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:30.238211 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:30.238218 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:30.238231 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:30.238790 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:30.238866 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:30.238896 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:30.239036 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:30.239265 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 03:04:30.239505 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":331,"threads_started":1}
I20260620 03:04:30.241207 22608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819: Bootstrap starting.
I20260620 03:04:30.244336 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:04:30.244834 22608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:30.245913 22608 log.cc:824] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:30.248968 22608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819: No bootstrap required, opened a new log
I20260620 03:04:30.256873 22611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:30.248849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46843 (local address 127.0.97.62:33481)
0620 03:04:30.249062 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:30.249070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:30.249451 (+ 381us) server_negotiation.cc:407] Connection header received
0620 03:04:30.249743 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:30.249749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:30.249831 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:30.249943 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:30.250724 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:30.251957 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:30.254168 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:30.254690 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:30.254803 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:30.256068 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:30.256073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:30.256077 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:30.256112 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:30.256154 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:30.256170 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:30.256242 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:30.256417 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 03:04:30.256499 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 03:04:30.259434 22608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d4b433991c4bffabb4352abc1df819" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33481 } }
I20260620 03:04:30.259876 22608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:04:30.259981 22608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d4b433991c4bffabb4352abc1df819, State: Initialized, Role: FOLLOWER
I20260620 03:04:30.260560 22608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d4b433991c4bffabb4352abc1df819" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33481 } }
I20260620 03:04:30.260829 22608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:04:30.260982 22608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:04:30.261189 22608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:30.263455 22608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d4b433991c4bffabb4352abc1df819" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33481 } }
I20260620 03:04:30.263900 22608 leader_election.cc:302] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63d4b433991c4bffabb4352abc1df819; no voters:
I20260620 03:04:30.264421 22608 leader_election.cc:288] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:04:30.264568 22618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:04:30.265232 22618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819 [term 1 LEADER]: Becoming Leader. State: Replica: 63d4b433991c4bffabb4352abc1df819, State: Running, Role: LEADER
I20260620 03:04:30.265810 22618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d4b433991c4bffabb4352abc1df819" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33481 } }
I20260620 03:04:30.266642 22608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:30.268847 22619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63d4b433991c4bffabb4352abc1df819" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d4b433991c4bffabb4352abc1df819" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33481 } } }
I20260620 03:04:30.268889 22620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63d4b433991c4bffabb4352abc1df819. Latest consensus state: current_term: 1 leader_uuid: "63d4b433991c4bffabb4352abc1df819" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d4b433991c4bffabb4352abc1df819" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33481 } } }
I20260620 03:04:30.269215 22619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:30.269269 22620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:30.271500 22626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:04:30.278016 22626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:04:30.287657 22626 catalog_manager.cc:1347] Generated new cluster ID: 83073d94fc4741f39992660705c9d687
I20260620 03:04:30.287830 22626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:04:30.307097 22626 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:04:30.308661 22626 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:04:30.315991 22626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63d4b433991c4bffabb4352abc1df819: Generated new TSK 0
I20260620 03:04:30.316802 22626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:04:30.332741 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 22541
W20260620 03:04:30.333884 22603 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60829
W20260620 03:04:30.334528 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 22541: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:04:30.334663 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 22541
2026-06-20T03:04:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1924 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:04:30 dist-test-slave-f1j0 krb5kdc[22639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:04:30 dist-test-slave-f1j0 krb5kdc[22639](info): set up 2 sockets
Jun 20 03:04:30 dist-test-slave-f1j0 krb5kdc[22639](info): commencing operation
krb5kdc: starting...
W20260620 03:04:31.447225 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:04:31 dist-test-slave-f1j0 krb5kdc[22639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:04:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:04:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:04:31.712145 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38451
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:04:32.036021 22652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:32.036307 22652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:32.036485 22652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:32.043717 22652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:32.043828 22652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:32.043912 22652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:32.044016 22652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:32.044092 22652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:32.054009 22652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:32.055284 22652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:32.056600 22652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:32.068354 22658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:32.068382 22660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:32.068436 22657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:32.069368 22652 server_base.cc:1034] running on GCE node
I20260620 03:04:32.070181 22652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:32.071534 22652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:32.072768 22652 hybrid_clock.cc:648] HybridClock initialized: now 1781924672072705 us; error 53 us; skew 500 ppm
I20260620 03:04:32.073114 22652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:32 dist-test-slave-f1j0 krb5kdc[22639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:32.076129 22652 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:04:32.077497 22652 webserver.cc:460] Webserver started at http://127.0.97.62:38675/ using document root <none> and password file <none>
I20260620 03:04:32.078198 22652 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:32.078338 22652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:32.078649 22652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:32.081501 22652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "58bf539c08bc4be2a3821a5e22a44ff4"
format_stamp: "Formatted at 2026-06-20 03:04:32 on dist-test-slave-f1j0"
server_key: "3ab1ca4c9d27497bca2df6298054d926"
server_key_iv: "ecd81cd1781f8c4edbb931bf3b87b0ed"
server_key_version: "encryptionkey@0"
I20260620 03:04:32.082273 22652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "58bf539c08bc4be2a3821a5e22a44ff4"
format_stamp: "Formatted at 2026-06-20 03:04:32 on dist-test-slave-f1j0"
server_key: "3ab1ca4c9d27497bca2df6298054d926"
server_key_iv: "ecd81cd1781f8c4edbb931bf3b87b0ed"
server_key_version: "encryptionkey@0"
I20260620 03:04:32.087167 22652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:04:32.090512 22667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:32.091728 22652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:04:32.091902 22652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "58bf539c08bc4be2a3821a5e22a44ff4"
format_stamp: "Formatted at 2026-06-20 03:04:32 on dist-test-slave-f1j0"
server_key: "3ab1ca4c9d27497bca2df6298054d926"
server_key_iv: "ecd81cd1781f8c4edbb931bf3b87b0ed"
server_key_version: "encryptionkey@0"
I20260620 03:04:32.092092 22652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:32.097712 22652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:32.101686 22652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:32.101987 22652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:32.126471 22652 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38451
I20260620 03:04:32.126513 22718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38451 every 8 connection(s)
I20260620 03:04:32.127933 22652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:04:32.131769 22719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:32.136575 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 22652
I20260620 03:04:32.136763 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:04:32.137320 388 external_mini_cluster.cc:1428] Setting key 109be066b70d6351e007dc03aa7ef30c
I20260620 03:04:32.143150 22719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4: Bootstrap starting.
I20260620 03:04:32.144910 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:32.139065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50637 (local address 127.0.97.62:38451)
0620 03:04:32.139892 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:32.139904 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:32.139950 (+ 46us) server_negotiation.cc:407] Connection header received
0620 03:04:32.141720 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:32.141744 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:32.142070 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:32.142392 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:32.143410 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:32.143419 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:32.143432 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:32.143446 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:32.143508 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:32.143526 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:32.143689 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:32.143820 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 03:04:32.144015 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":371,"threads_started":1}
I20260620 03:04:32.146642 22719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:32.147881 22719 log.cc:824] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:32.148792 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:04:32.150970 22719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4: No bootstrap required, opened a new log
I20260620 03:04:32.156111 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:32.153805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33841 (local address 127.0.97.62:38451)
0620 03:04:32.154011 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:32.154017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:32.154340 (+ 323us) server_negotiation.cc:407] Connection header received
0620 03:04:32.154493 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:32.154497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:32.154543 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:32.154623 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:32.155492 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:32.155496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:32.155502 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 03:04:32.155515 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:32.155549 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 03:04:32.155563 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:32.155650 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:32.155788 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 03:04:32.155831 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 03:04:32.160504 22719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58bf539c08bc4be2a3821a5e22a44ff4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38451 } }
I20260620 03:04:32.160895 22719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 03:04:32.160993 22719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58bf539c08bc4be2a3821a5e22a44ff4, State: Initialized, Role: FOLLOWER
I20260620 03:04:32.161671 22719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58bf539c08bc4be2a3821a5e22a44ff4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38451 } }
I20260620 03:04:32.161906 22719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 03:04:32.162056 22719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 03:04:32.162217 22719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:32.164590 22719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58bf539c08bc4be2a3821a5e22a44ff4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38451 } }
I20260620 03:04:32.165087 22719 leader_election.cc:302] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58bf539c08bc4be2a3821a5e22a44ff4; no voters:
I20260620 03:04:32.165696 22719 leader_election.cc:288] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 03:04:32.165827 22729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:04:32.166481 22729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4 [term 1 LEADER]: Becoming Leader. State: Replica: 58bf539c08bc4be2a3821a5e22a44ff4, State: Running, Role: LEADER
I20260620 03:04:32.167040 22729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58bf539c08bc4be2a3821a5e22a44ff4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38451 } }
I20260620 03:04:32.167716 22719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:32.169907 22731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58bf539c08bc4be2a3821a5e22a44ff4. Latest consensus state: current_term: 1 leader_uuid: "58bf539c08bc4be2a3821a5e22a44ff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58bf539c08bc4be2a3821a5e22a44ff4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38451 } } }
I20260620 03:04:32.169997 22730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58bf539c08bc4be2a3821a5e22a44ff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58bf539c08bc4be2a3821a5e22a44ff4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38451 } } }
I20260620 03:04:32.170390 22730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:32.170284 22731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:32.172519 22735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:04:32.177645 22735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:04:32.186909 22735 catalog_manager.cc:1347] Generated new cluster ID: 0d09a2cb27534deb979f3f4ea312efa4
I20260620 03:04:32.187039 22735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:04:32.198503 22735 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:04:32.199075 22735 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:04:32.207294 22735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58bf539c08bc4be2a3821a5e22a44ff4: Generated new TSK 0
I20260620 03:04:32.208181 22735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:04:32.223295 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 22652
W20260620 03:04:32.224356 22714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50637
W20260620 03:04:32.225171 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 22652: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 03:04:32.225337 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 22652
2026-06-20T03:04:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1890 ms)
[----------] 18 tests from AuthTokenIssuingTest (34869 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:04:32 dist-test-slave-f1j0 krb5kdc[22750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:04:32 dist-test-slave-f1j0 krb5kdc[22750](info): set up 2 sockets
Jun 20 03:04:32 dist-test-slave-f1j0 krb5kdc[22750](info): commencing operation
krb5kdc: starting...
W20260620 03:04:33.322160 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:04:33 dist-test-slave-f1j0 krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:04:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:04:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:04:33.584581 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45469
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45469,127.0.97.61:38173,127.0.97.60:37171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:04:33.915567 22763 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 03:04:33.915885 22763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:33.915957 22763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:33.916038 22763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:33.923156 22763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:33.923241 22763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:33.923300 22763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:33.923352 22763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:33.923404 22763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:33.933075 22763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45469,127.0.97.61:38173,127.0.97.60:37171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:33.934301 22763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:33.935717 22763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:33.947383 22769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:33.947459 22771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:33.947410 22768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:33.948220 22763 server_base.cc:1034] running on GCE node
I20260620 03:04:33.948942 22763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:33.950259 22763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:33.951473 22763 hybrid_clock.cc:648] HybridClock initialized: now 1781924673951417 us; error 27 us; skew 500 ppm
I20260620 03:04:33.951788 22763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:33 dist-test-slave-f1j0 krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:33.954886 22763 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:04:33.956272 22763 webserver.cc:460] Webserver started at http://127.0.97.62:36749/ using document root <none> and password file <none>
I20260620 03:04:33.956938 22763 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:33.957060 22763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:33.957371 22763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:33.960306 22763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "67c809f67c2541229807729246a58e72"
format_stamp: "Formatted at 2026-06-20 03:04:33 on dist-test-slave-f1j0"
server_key: "39d1aec45368f1fae7b4e15318c39696"
server_key_iv: "d2890415c1b39115e68c3ab05723fc22"
server_key_version: "encryptionkey@0"
I20260620 03:04:33.961056 22763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "67c809f67c2541229807729246a58e72"
format_stamp: "Formatted at 2026-06-20 03:04:33 on dist-test-slave-f1j0"
server_key: "39d1aec45368f1fae7b4e15318c39696"
server_key_iv: "d2890415c1b39115e68c3ab05723fc22"
server_key_version: "encryptionkey@0"
I20260620 03:04:33.966130 22763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260620 03:04:33.969497 22778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:33.970851 22763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:04:33.971019 22763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "67c809f67c2541229807729246a58e72"
format_stamp: "Formatted at 2026-06-20 03:04:33 on dist-test-slave-f1j0"
server_key: "39d1aec45368f1fae7b4e15318c39696"
server_key_iv: "d2890415c1b39115e68c3ab05723fc22"
server_key_version: "encryptionkey@0"
I20260620 03:04:33.971233 22763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:33.986341 22763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:33.990805 22763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:33.991074 22763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:34.009117 22763 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45469
I20260620 03:04:34.009140 22829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45469 every 8 connection(s)
I20260620 03:04:34.010604 22763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:04:34.014276 22830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:34.019819 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 22763
I20260620 03:04:34.020018 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:04:34.020601 388 external_mini_cluster.cc:1428] Setting key 13fb84ee7942dbd0cd9ecb7932e9bcbc
I20260620 03:04:34.020828 22830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:04:34 dist-test-slave-f1j0 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260620 03:04:34.044974 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.025064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43928 (local address 127.0.97.62:45469)
0620 03:04:34.025780 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:34.025791 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:34.025846 (+ 55us) server_negotiation.cc:407] Connection header received
0620 03:04:34.027264 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:34.027275 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:34.027582 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:34.027805 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:34.029288 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:34.030533 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:34.032852 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:34.033416 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:34.033552 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:34.037690 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:34.037701 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:34.037715 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:34.037743 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:34.040578 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:34.041268 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:34.041279 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:34.041282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:34.041432 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:34.041841 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:34.041849 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:34.041852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:34.042360 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:34.042572 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:34.043093 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 03:04:34.043484 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":461,"thread_start_us":301,"threads_started":1}
I20260620 03:04:34.045009 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.023767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45469 (local address 127.0.0.1:43928)
0620 03:04:34.024587 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.024635 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 03:04:34.026842 (+ 2207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:04:34.027830 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:04:34.027835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:04:34.028553 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:04:34.029055 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:34.029084 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:34.030688 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:34.030696 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:34.032705 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:34.032722 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:34.033674 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:34.033683 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:34.033947 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:34.034646 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:04:34.034687 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:04:34.037433 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:04:34.040719 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:34.040733 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:34.040746 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:34.041139 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:34.041538 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:34.041544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:34.041547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:34.041739 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:34.042498 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:04:34.042505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:04:34.042727 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 03:04:34.042962 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 03:04:34.043594 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":159,"thread_start_us":349,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 03:04:34.048480 22830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:04:34.049767 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.049211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38173 (local address 127.0.0.1:49484)
0620 03:04:34.049419 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.049563 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 03:04:34.050074 22780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38173: connect: Connection refused (error 111)
I20260620 03:04:34.051805 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.051332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38173 (local address 127.0.0.1:49490)
0620 03:04:34.051513 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.051613 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 03:04:34.051995 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38173: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38173: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 03:04:34.077776 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38173
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45469,127.0.97.61:38173,127.0.97.60:37171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:04:34.101361 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } attempt: 1
I20260620 03:04:34.102561 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.102126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38173 (local address 127.0.0.1:49504)
0620 03:04:34.102336 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.102440 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 03:04:34.103519 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.103166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38173 (local address 127.0.0.1:49512)
0620 03:04:34.103332 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.103396 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 03:04:34.103709 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38173: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38173: connect: Connection refused (error 111)
I20260620 03:04:34.171993 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } attempt: 2
I20260620 03:04:34.173187 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.172689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38173 (local address 127.0.0.1:49518)
0620 03:04:34.172956 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.173051 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 03:04:34.174257 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.173893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38173 (local address 127.0.0.1:49534)
0620 03:04:34.174058 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.174124 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 03:04:34.174477 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38173: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38173: connect: Connection refused (error 111)
I20260620 03:04:34.265702 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } attempt: 3
I20260620 03:04:34.267396 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.266746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38173 (local address 127.0.0.1:49550)
0620 03:04:34.266988 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.267161 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 03:04:34.268642 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.268081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38173 (local address 127.0.0.1:49556)
0620 03:04:34.268305 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.268427 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 03:04:34.268929 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38173: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38173: connect: Connection refused (error 111)
W20260620 03:04:34.404758 22837 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 03:04:34.405074 22837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:34.405140 22837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:34.405228 22837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 03:04:34.412192 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } attempt: 4
W20260620 03:04:34.412791 22837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:34.412873 22837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:34.412966 22837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:34.413056 22837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:34.413143 22837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:34.413414 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.412825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38173 (local address 127.0.0.1:49566)
0620 03:04:34.413143 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.413262 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260620 03:04:34.414423 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.414089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38173 (local address 127.0.0.1:49576)
0620 03:04:34.414225 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.414302 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 03:04:34.414626 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38173: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38173: connect: Connection refused (error 111)
I20260620 03:04:34.423156 22837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45469,127.0.97.61:38173,127.0.97.60:37171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:34.424439 22837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:34.425810 22837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:34.437685 22843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:34.437772 22845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:34.437704 22842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:34.438118 22837 server_base.cc:1034] running on GCE node
I20260620 03:04:34.439170 22837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:34.440591 22837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:34.441823 22837 hybrid_clock.cc:648] HybridClock initialized: now 1781924674441760 us; error 69 us; skew 500 ppm
I20260620 03:04:34.442139 22837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:34 dist-test-slave-f1j0 krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:34.445240 22837 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 03:04:34.446715 22837 webserver.cc:460] Webserver started at http://127.0.97.61:35439/ using document root <none> and password file <none>
I20260620 03:04:34.447398 22837 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:34.447520 22837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:34.447829 22837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:34.450834 22837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-1/data/instance:
uuid: "9c626f87c3254818af032e7ca19cf222"
format_stamp: "Formatted at 2026-06-20 03:04:34 on dist-test-slave-f1j0"
server_key: "d72311b464d8490950bd2b1ae9cc971e"
server_key_iv: "58de13b4a21999be8be9b9b7402cfe1e"
server_key_version: "encryptionkey@0"
I20260620 03:04:34.451606 22837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-1/wal/instance:
uuid: "9c626f87c3254818af032e7ca19cf222"
format_stamp: "Formatted at 2026-06-20 03:04:34 on dist-test-slave-f1j0"
server_key: "d72311b464d8490950bd2b1ae9cc971e"
server_key_iv: "58de13b4a21999be8be9b9b7402cfe1e"
server_key_version: "encryptionkey@0"
I20260620 03:04:34.458941 22837 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260620 03:04:34.463056 22852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:34.464463 22837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 03:04:34.464658 22837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-1/wal
uuid: "9c626f87c3254818af032e7ca19cf222"
format_stamp: "Formatted at 2026-06-20 03:04:34 on dist-test-slave-f1j0"
server_key: "d72311b464d8490950bd2b1ae9cc971e"
server_key_iv: "58de13b4a21999be8be9b9b7402cfe1e"
server_key_version: "encryptionkey@0"
I20260620 03:04:34.464880 22837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:34.495925 22837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:34.499943 22837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:34.500209 22837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:34.518460 22837 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38173
I20260620 03:04:34.518484 22903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38173 every 8 connection(s)
I20260620 03:04:34.519804 22837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-1/data/info.pb
I20260620 03:04:34.521409 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 22837
I20260620 03:04:34.521629 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-1/wal/instance
I20260620 03:04:34.522090 388 external_mini_cluster.cc:1428] Setting key fd093b9e4ef263237a970130c3e6bd34
I20260620 03:04:34.523613 22904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:34.529910 22904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:04:34 dist-test-slave-f1j0 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260620 03:04:34.550190 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.532440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.0.97.62:45469)
0620 03:04:34.532675 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:34.532684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:34.533793 (+ 1109us) server_negotiation.cc:407] Connection header received
0620 03:04:34.535622 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:34.535632 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:34.535724 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:34.535853 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:34.537272 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:34.538463 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:34.539932 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:34.540346 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:34.540475 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:34.543894 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:34.543903 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:34.543911 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:34.543957 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:34.546892 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:34.547847 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:34.547851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:34.547854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:34.547964 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:34.548437 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:34.548440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:34.548442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:34.548709 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:34.548825 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:34.549795 (+ 970us) server_negotiation.cc:299] Negotiation successful
0620 03:04:34.549962 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 03:04:34.551738 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.532816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45469 (local address 127.0.0.1:43936)
0620 03:04:34.533607 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.533659 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 03:04:34.535360 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:04:34.536036 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:04:34.536044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:04:34.536538 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:04:34.537094 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:34.537110 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:34.538609 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:34.538613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:34.539780 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:34.539797 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:34.540581 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:34.540589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:34.540814 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:34.541482 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:04:34.541505 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:04:34.543651 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:04:34.547034 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:34.547045 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:34.547066 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:34.547740 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:34.548067 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:34.548075 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:34.548082 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:34.548335 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:34.548856 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:04:34.548867 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:04:34.549368 (+ 501us) client_negotiation.cc:769] Sending connection context
0620 03:04:34.549721 (+ 353us) client_negotiation.cc:240] Negotiation successful
0620 03:04:34.550289 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":341,"threads_started":1}
I20260620 03:04:34.553474 22904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:04:34 dist-test-slave-f1j0 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:04:34.567212 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.554359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38173 (local address 127.0.0.1:49586)
0620 03:04:34.554543 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.554560 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 03:04:34.554651 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:04:34.555578 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:04:34.555583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:04:34.556001 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:04:34.556124 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:34.556132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:34.557489 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:34.557494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:34.558361 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:34.558371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:34.559142 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:34.559146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:34.559358 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:34.559913 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:04:34.559929 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:04:34.561876 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:04:34.564816 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:34.564824 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:34.564828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:34.565167 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:34.565498 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:34.565505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:34.565508 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:34.565689 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:34.566251 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:04:34.566263 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:04:34.566421 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 03:04:34.566545 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 03:04:34.566799 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":36}
I20260620 03:04:34.567277 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.554566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49586 (local address 127.0.97.61:38173)
0620 03:04:34.555198 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:34.555203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:34.555225 (+ 22us) server_negotiation.cc:407] Connection header received
0620 03:04:34.555266 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:34.555273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:34.555445 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:34.555591 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:34.556287 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:34.557351 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:34.558492 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:34.559009 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:34.559136 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:34.562069 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:34.562079 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:34.562091 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:34.562116 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:34.564693 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:34.565282 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:34.565291 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:34.565295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:34.565404 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:34.565786 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:34.565791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:34.565795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:34.566116 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:34.566321 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:34.566682 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 03:04:34.566894 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":482,"thread_start_us":357,"threads_started":1}
I20260620 03:04:34.569876 22904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:04:34.571115 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.570585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42364)
0620 03:04:34.570793 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.570953 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 03:04:34.571350 22855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
I20260620 03:04:34.572691 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.572410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42378)
0620 03:04:34.572528 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.572601 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 03:04:34.572993 22904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 03:04:34.580322 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37171
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45469,127.0.97.61:38173,127.0.97.60:37171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:04:34.622370 22904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } attempt: 1
I20260620 03:04:34.623725 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.623163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42392)
0620 03:04:34.623398 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.623508 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 03:04:34.624787 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.624393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42396)
0620 03:04:34.624565 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.624654 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 03:04:34.625013 22904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
I20260620 03:04:34.693265 22904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } attempt: 2
I20260620 03:04:34.694635 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.694040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42408)
0620 03:04:34.694301 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.694420 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 03:04:34.695614 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.695219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42414)
0620 03:04:34.695373 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.695458 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 03:04:34.695843 22904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
I20260620 03:04:34.713865 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } attempt: 5
Jun 20 03:04:34 dist-test-slave-f1j0 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:04:34.726805 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.714770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.0.97.61:38173)
0620 03:04:34.714954 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:34.714959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:34.714981 (+ 22us) server_negotiation.cc:407] Connection header received
0620 03:04:34.715151 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:34.715156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:34.715243 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:34.715358 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:34.716124 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:34.716820 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:34.718048 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:34.718485 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:34.718619 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:34.721727 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:34.721732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:34.721735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:34.721773 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:34.724312 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:34.724928 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:34.724938 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:34.724941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:34.725050 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:34.725411 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:34.725418 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:34.725421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:34.725754 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:34.725907 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:34.726261 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 03:04:34.726462 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 03:04:34.726795 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.714608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38173 (local address 127.0.0.1:49600)
0620 03:04:34.714846 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.714869 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 03:04:34.715003 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:04:34.715396 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:04:34.715402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:04:34.715805 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:04:34.715976 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:34.715985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:34.716960 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:34.716965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:34.717905 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:34.717917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:34.718632 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:34.718638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:34.718847 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:34.719497 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:04:34.719514 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:04:34.721511 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:04:34.724449 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:34.724454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:34.724458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:34.724813 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:34.725164 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:34.725168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:34.725172 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:34.725312 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:34.725869 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:04:34.725874 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:04:34.726023 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 03:04:34.726161 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 03:04:34.726449 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 03:04:34.728057 22830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:04:34.728983 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.728623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42430)
0620 03:04:34.728786 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.728852 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 03:04:34.729840 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.729511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42446)
0620 03:04:34.729668 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.729746 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 03:04:34.730053 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
I20260620 03:04:34.781267 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } attempt: 1
I20260620 03:04:34.782532 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.781942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42462)
0620 03:04:34.782191 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.782319 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 03:04:34.783560 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.783194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42476)
0620 03:04:34.783342 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.783427 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 03:04:34.783797 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
I20260620 03:04:34.787073 22904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } attempt: 3
I20260620 03:04:34.788174 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.787604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42480)
0620 03:04:34.787843 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.787956 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 03:04:34.789247 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.788741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42482)
0620 03:04:34.789008 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.789108 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
W20260620 03:04:34.789446 22904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
I20260620 03:04:34.852021 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } attempt: 2
I20260620 03:04:34.853269 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.852658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42488)
0620 03:04:34.852919 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.853035 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 03:04:34.854393 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.853966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42492)
0620 03:04:34.854138 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.854235 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 03:04:34.854655 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
W20260620 03:04:34.913702 22911 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 03:04:34.914045 22911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:34.914160 22911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:34.914285 22911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:34.921324 22911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:34.921406 22911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:34.921509 22911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:34.921609 22911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:34.921710 22911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:34.931335 22911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45469,127.0.97.61:38173,127.0.97.60:37171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:34.932583 22911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:34.932677 22904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } attempt: 4
I20260620 03:04:34.933905 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.933336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42506)
0620 03:04:34.933594 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.933702 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 03:04:34.934317 22911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 03:04:34.934894 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.934531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42514)
0620 03:04:34.934683 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.934766 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 03:04:34.935143 22904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
W20260620 03:04:34.946192 22916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:34.946146 22917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:34.946223 22919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:34.947184 22911 server_base.cc:1034] running on GCE node
I20260620 03:04:34.947925 22911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:34.949211 22911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:34.950418 22911 hybrid_clock.cc:648] HybridClock initialized: now 1781924674950364 us; error 29 us; skew 500 ppm
I20260620 03:04:34.950767 22911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:34 dist-test-slave-f1j0 krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:34.953890 22911 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 03:04:34.955264 22911 webserver.cc:460] Webserver started at http://127.0.97.60:33451/ using document root <none> and password file <none>
I20260620 03:04:34.955919 22911 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:34.956027 22911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:34.956341 22911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:34.959232 22911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-2/data/instance:
uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b"
format_stamp: "Formatted at 2026-06-20 03:04:34 on dist-test-slave-f1j0"
server_key: "223bf7ca95f711b95b5b21163a6040d0"
server_key_iv: "1fce586ec8a5378cca781412c4530959"
server_key_version: "encryptionkey@0"
I20260620 03:04:34.959968 22911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-2/wal/instance:
uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b"
format_stamp: "Formatted at 2026-06-20 03:04:34 on dist-test-slave-f1j0"
server_key: "223bf7ca95f711b95b5b21163a6040d0"
server_key_iv: "1fce586ec8a5378cca781412c4530959"
server_key_version: "encryptionkey@0"
I20260620 03:04:34.960857 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } attempt: 3
I20260620 03:04:34.961958 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.961360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42522)
0620 03:04:34.961626 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.961724 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 03:04:34.963083 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:34.962651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42532)
0620 03:04:34.962861 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:34.962942 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260620 03:04:34.963344 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37171: connect: Connection refused (error 111)
I20260620 03:04:34.965446 22911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 03:04:34.968781 22926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:34.970256 22911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 03:04:34.970431 22911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-2/wal
uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b"
format_stamp: "Formatted at 2026-06-20 03:04:34 on dist-test-slave-f1j0"
server_key: "223bf7ca95f711b95b5b21163a6040d0"
server_key_iv: "1fce586ec8a5378cca781412c4530959"
server_key_version: "encryptionkey@0"
I20260620 03:04:34.970655 22911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:35.005043 22911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:35.009707 22911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:35.009985 22911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:35.028578 22911 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:37171
I20260620 03:04:35.028585 22977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:37171 every 8 connection(s)
I20260620 03:04:35.030043 22911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-2/data/info.pb
I20260620 03:04:35.033617 22978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:35.035012 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 22911
I20260620 03:04:35.035204 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781924320356866-388-0/minicluster-data/master-2/wal/instance
I20260620 03:04:35.035607 388 external_mini_cluster.cc:1428] Setting key 0811dde0bfdd3b9371710b3c104a6afa
I20260620 03:04:35.042521 22978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:04:35 dist-test-slave-f1j0 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 03:04:35 dist-test-slave-f1j0 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:04:35.058609 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:35.037384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42540 (local address 127.0.97.60:37171)
0620 03:04:35.038194 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.038205 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.038246 (+ 41us) server_negotiation.cc:407] Connection header received
0620 03:04:35.039731 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.039755 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.040133 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.040446 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.042117 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.043450 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.045061 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.046140 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.046319 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:35.050574 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:35.050581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:35.050590 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:35.050628 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:35.054180 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.054927 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.054937 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.054948 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.055116 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.055516 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.055521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.055525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.056208 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:35.056519 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:35.056891 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 03:04:35.057297 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":412,"threads_started":1}
I20260620 03:04:35.060735 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:35.045072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.0.97.62:45469)
0620 03:04:35.045249 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.045253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.045890 (+ 637us) server_negotiation.cc:407] Connection header received
0620 03:04:35.046072 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.046078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.046148 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.046251 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.047528 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.048689 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.050060 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.050671 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.050769 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:35.054640 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:35.054645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:35.054648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:35.054680 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:35.058119 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.058786 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.058791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.058794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.058964 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.059340 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.059345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.059353 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.059592 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:35.059752 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:35.060169 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 03:04:35.060376 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 03:04:35.060773 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:35.045043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45469 (local address 127.0.0.1:43952)
0620 03:04:35.045767 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:35.045805 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 03:04:35.045953 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:04:35.046318 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:04:35.046325 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:04:35.047088 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:04:35.047336 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:35.047344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.048856 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.048860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:35.049923 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:35.049934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.050847 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.050853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:35.051065 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:35.051769 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:04:35.051801 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:04:35.054468 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:04:35.058235 (+ 3767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:35.058244 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:35.058248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:35.058642 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:35.059069 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:35.059075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:35.059078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:35.059224 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:35.059714 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:04:35.059723 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:04:35.059893 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 03:04:35.060049 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 03:04:35.060360 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":351,"threads_started":1}
I20260620 03:04:35.062260 22978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:04:35 dist-test-slave-f1j0 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 03:04:35 dist-test-slave-f1j0 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:04:35.076793 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:35.061457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43966 (local address 127.0.97.62:45469)
0620 03:04:35.061655 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.061663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.061681 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:04:35.061864 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.061868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.061918 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.061989 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.063250 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.063943 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.065273 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.065758 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.065848 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:35.070401 (+ 4553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:35.070407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:35.070417 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:35.070456 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:35.073858 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.074700 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.074708 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.074714 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.074882 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.075322 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.075327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.075330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.075658 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:35.075935 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:35.076248 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 03:04:35.076470 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 03:04:35.077016 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:35.062998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38173 (local address 127.0.0.1:49612)
0620 03:04:35.063321 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:35.063334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 03:04:35.063437 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:04:35.063742 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:04:35.063746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:04:35.064187 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:04:35.064365 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:35.064375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.065318 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.065324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:35.066521 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:35.066536 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.067237 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.067244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:35.067488 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:35.067971 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:04:35.067997 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:04:35.070513 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:04:35.074125 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:35.074131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:35.074135 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:35.074654 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:35.075120 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:35.075125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:35.075128 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:35.075311 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:35.075937 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:04:35.075941 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:04:35.076072 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 03:04:35.076217 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 03:04:35.076498 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213}
I20260620 03:04:35.076893 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:35.063012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49612 (local address 127.0.97.61:38173)
0620 03:04:35.063182 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.063190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.063394 (+ 204us) server_negotiation.cc:407] Connection header received
0620 03:04:35.063557 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.063562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.063630 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.063725 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.064501 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.065148 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.066669 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.067082 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.067184 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:35.070694 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:35.070701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:35.070707 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:35.070747 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:35.074001 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.074866 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.074871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.074874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.075018 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.075415 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.075418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.075420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.075679 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:35.075877 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:35.076329 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 03:04:35.076528 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 03:04:35.078106 22978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:04:35 dist-test-slave-f1j0 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 03:04:35 dist-test-slave-f1j0 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:04:35.090850 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:35.078831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42546 (local address 127.0.97.60:37171)
0620 03:04:35.078973 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.078976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.078990 (+ 14us) server_negotiation.cc:407] Connection header received
0620 03:04:35.079119 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.079122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.079176 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.079277 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.079908 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.080561 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.081683 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.082100 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.082239 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:35.085487 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:35.085492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:35.085495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:35.085520 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:35.088208 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.088921 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.088928 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.088932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.089037 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.089365 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.089372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.089375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.089699 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:35.089838 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:35.090140 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 03:04:35.090537 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":167,"server-negotiator.queue_time_us":41}
I20260620 03:04:35.090850 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:35.078792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42546)
0620 03:04:35.078920 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:35.078938 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:04:35.079023 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:04:35.079292 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:04:35.079296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:04:35.079662 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:04:35.079787 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:35.079795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.080684 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.080691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:35.081506 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:35.081516 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.082253 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.082258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:35.082471 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:35.083045 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:04:35.083066 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:04:35.085337 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:04:35.088339 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:35.088344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:35.088348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:35.088802 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:35.089141 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:35.089147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:35.089151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:35.089272 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:35.089804 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:04:35.089808 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:04:35.089938 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 03:04:35.090070 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 03:04:35.090399 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":24}
I20260620 03:04:35.092299 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:35.078472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.0.97.61:38173)
0620 03:04:35.079025 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.079031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.079047 (+ 16us) server_negotiation.cc:407] Connection header received
0620 03:04:35.079096 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.079101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.079165 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.079265 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.080363 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.081045 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.082309 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.082849 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.082971 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:35.087179 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:35.087183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:35.087186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:35.087215 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:35.089684 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.090316 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.090321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.090325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.090460 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.090852 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.090857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.090861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.091210 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:35.091462 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:35.091773 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 03:04:35.091988 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449}
I20260620 03:04:35.096042 22978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b: Bootstrap starting.
I20260620 03:04:35.096771 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:04:35.099392 22978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:35.100419 22978 log.cc:824] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:35.099326 22805 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "67c809f67c2541229807729246a58e72"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43966
I20260620 03:04:35.103189 22978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b: No bootstrap required, opened a new log
I20260620 03:04:35.110117 22978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c809f67c2541229807729246a58e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } peers { permanent_uuid: "9c626f87c3254818af032e7ca19cf222" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } } peers { permanent_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } }
I20260620 03:04:35.110453 22978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1f8eeafc734a9dbe97d6d8fa093a1b, State: Initialized, Role: FOLLOWER
I20260620 03:04:35.110881 22978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c809f67c2541229807729246a58e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } peers { permanent_uuid: "9c626f87c3254818af032e7ca19cf222" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } } peers { permanent_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } }
I20260620 03:04:35.111774 22984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c809f67c2541229807729246a58e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } peers { permanent_uuid: "9c626f87c3254818af032e7ca19cf222" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } } peers { permanent_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } } }
I20260620 03:04:35.112172 22984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:04:35.112753 22978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b [sys.catalog]: configured and running, proceeding with master startup.
W20260620 03:04:35.122274 22995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 03:04:35.122426 22995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 03:04:35.120015 22911 master_runner.cc:186] Error getting master registration for 127.0.97.62:45469: OK, instance_id { permanent_uuid: "67c809f67c2541229807729246a58e72" instance_seqno: 1781924674003983 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 03:04:35.127959 22911 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 03:04:35.140583 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } attempt: 4
Jun 20 03:04:35 dist-test-slave-f1j0 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 03:04:35.157167 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:35.141362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42552)
0620 03:04:35.141632 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:35.141656 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 03:04:35.141799 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:04:35.142178 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:04:35.142183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:04:35.142706 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:04:35.142936 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:35.142949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.144448 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.144455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:35.145994 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:35.146010 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.147045 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.147052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:35.147245 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:35.147924 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:04:35.147948 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:04:35.150845 (+ 2897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:04:35.154201 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:35.154208 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:35.154212 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:35.154710 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:35.155175 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:35.155180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:35.155184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:35.155384 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:35.156116 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:04:35.156121 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:04:35.156296 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 03:04:35.156445 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 03:04:35.156764 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 03:04:35.157169 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:35.141435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42552 (local address 127.0.97.60:37171)
0620 03:04:35.141658 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.141669 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.141730 (+ 61us) server_negotiation.cc:407] Connection header received
0620 03:04:35.141927 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.141933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.142012 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.142121 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.143099 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.144282 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.146129 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.146886 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.147017 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:35.151072 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:35.151081 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:35.151085 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:35.151121 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:35.154082 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.154843 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.154867 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.154872 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.155021 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.155523 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.155531 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.155535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.155978 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:35.156162 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:35.156563 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 03:04:35.156800 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:04:35.163507 22830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72: Bootstrap starting.
I20260620 03:04:35.166733 22830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:35.167779 22830 log.cc:824] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:35.171247 22830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72: No bootstrap required, opened a new log
I20260620 03:04:35.179049 22830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c809f67c2541229807729246a58e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } peers { permanent_uuid: "9c626f87c3254818af032e7ca19cf222" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } } peers { permanent_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } }
I20260620 03:04:35.179502 22830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67c809f67c2541229807729246a58e72, State: Initialized, Role: FOLLOWER
I20260620 03:04:35.180043 22830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c809f67c2541229807729246a58e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } peers { permanent_uuid: "9c626f87c3254818af032e7ca19cf222" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } } peers { permanent_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } }
I20260620 03:04:35.181063 22998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c809f67c2541229807729246a58e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } peers { permanent_uuid: "9c626f87c3254818af032e7ca19cf222" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } } peers { permanent_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } } }
I20260620 03:04:35.181468 22998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:04:35.182402 22830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:35.182586 22805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 03:04:35.182776 22805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:35.185547 22805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c809f67c2541229807729246a58e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } peers { permanent_uuid: "9c626f87c3254818af032e7ca19cf222" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } } peers { permanent_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } }
I20260620 03:04:35.187752 22953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "67c809f67c2541229807729246a58e72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b"
I20260620 03:04:35.188230 22953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:35.191644 22953 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67c809f67c2541229807729246a58e72 in term 1.
I20260620 03:04:35.192308 22781 leader_election.cc:302] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d1f8eeafc734a9dbe97d6d8fa093a1b, 67c809f67c2541229807729246a58e72; no voters:
I20260620 03:04:35.192782 22998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:04:35.193511 22998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [term 1 LEADER]: Becoming Leader. State: Replica: 67c809f67c2541229807729246a58e72, State: Running, Role: LEADER
I20260620 03:04:35.194083 22998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c809f67c2541229807729246a58e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } peers { permanent_uuid: "9c626f87c3254818af032e7ca19cf222" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } } peers { permanent_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } }
I20260620 03:04:35.196226 22805 leader_election.cc:288] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [CANDIDATE]: Term 1 election: Requested vote from peers 9c626f87c3254818af032e7ca19cf222 (127.0.97.61:38173), 5d1f8eeafc734a9dbe97d6d8fa093a1b (127.0.97.60:37171)
I20260620 03:04:35.200309 23004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67c809f67c2541229807729246a58e72. Latest consensus state: current_term: 1 leader_uuid: "67c809f67c2541229807729246a58e72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c809f67c2541229807729246a58e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } peers { permanent_uuid: "9c626f87c3254818af032e7ca19cf222" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } } peers { permanent_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } } }
I20260620 03:04:35.201248 23004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:35.222190 23008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:04:35.234351 22904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } attempt: 5
W20260620 03:04:35.243917 22784 tablet.cc:2369] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 03:04:35 dist-test-slave-f1j0 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260620 03:04:35.262235 22932 tablet.cc:2369] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:04:35.277208 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:35.235047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37171 (local address 127.0.0.1:42556)
0620 03:04:35.235419 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:35.235444 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 03:04:35.235594 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:04:35.236341 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:04:35.236350 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:04:35.237062 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:04:35.237366 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:35.237385 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.239521 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.239530 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:35.241495 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:35.241517 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.243416 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.243426 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:35.243965 (+ 539us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:35.245181 (+ 1216us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:04:35.245242 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:04:35.250492 (+ 5250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:04:35.263265 (+ 12773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:35.263271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:35.263275 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:35.264529 (+ 1254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:35.267047 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:35.267088 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:35.267093 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:35.267413 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:35.274840 (+ 7427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:04:35.274849 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:04:35.275178 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 03:04:35.275596 (+ 418us) client_negotiation.cc:240] Negotiation successful
0620 03:04:35.276234 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260620 03:04:35.280277 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:35.235324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42556 (local address 127.0.97.60:37171)
0620 03:04:35.235567 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.235574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.235597 (+ 23us) server_negotiation.cc:407] Connection header received
0620 03:04:35.235771 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.235780 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.235893 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.236038 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.237631 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.239253 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.241838 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.242894 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.243055 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:35.250847 (+ 7792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:35.250855 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:35.250860 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:35.250913 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:35.262447 (+ 11534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.265479 (+ 3032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.265490 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.265495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.266489 (+ 994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.267899 (+ 1410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.267906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.267910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.274399 (+ 6489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:35.277735 (+ 3336us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:35.278546 (+ 811us) server_negotiation.cc:299] Negotiation successful
0620 03:04:35.279480 (+ 934us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 03:04:35.288293 22904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222: Bootstrap starting.
I20260620 03:04:35.292783 22904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:35.293536 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:35.230010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.0.97.62:45469)
0620 03:04:35.230280 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.230290 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.247570 (+ 17280us) server_negotiation.cc:407] Connection header received
0620 03:04:35.248182 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.248192 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.248311 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.248405 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.265176 (+ 16771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.266457 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.274094 (+ 7637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.277349 (+ 3255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.277506 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:35.285979 (+ 8473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:35.285992 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:35.286000 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:35.286046 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:35.289880 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.290982 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.290988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.290992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.291209 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.291648 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.291653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.291657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.291996 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:35.292260 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:35.292814 (+ 554us) server_negotiation.cc:299] Negotiation successful
0620 03:04:35.293064 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 03:04:35.293697 23008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:04:35.294641 22904 log.cc:824] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:35.296542 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:35.232552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49628 (local address 127.0.97.61:38173)
0620 03:04:35.232801 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.232810 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.251507 (+ 18697us) server_negotiation.cc:407] Connection header received
0620 03:04:35.270035 (+ 18528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.270042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.270137 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.270215 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.274166 (+ 3951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.275290 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.277427 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.278133 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.278229 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:35.284154 (+ 5925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:35.284161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:35.284166 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:35.284207 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:35.288116 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.288993 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.288999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.289002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.289140 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.289648 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.289654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.289659 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.290020 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:35.290302 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:35.295955 (+ 5653us) server_negotiation.cc:299] Negotiation successful
0620 03:04:35.296195 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 03:04:35.301298 22904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222: No bootstrap required, opened a new log
I20260620 03:04:35.312849 22904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c809f67c2541229807729246a58e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } peers { permanent_uuid: "9c626f87c3254818af032e7ca19cf222" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } } peers { permanent_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } }
I20260620 03:04:35.313251 22904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c626f87c3254818af032e7ca19cf222, State: Initialized, Role: FOLLOWER
I20260620 03:04:35.313989 22904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c809f67c2541229807729246a58e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } peers { permanent_uuid: "9c626f87c3254818af032e7ca19cf222" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } } peers { permanent_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } }
I20260620 03:04:35.318339 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:35.252087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.0.97.60:37171)
0620 03:04:35.273893 (+ 21806us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.273900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.273927 (+ 27us) server_negotiation.cc:407] Connection header received
0620 03:04:35.273983 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.273990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.275060 (+ 1070us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.275177 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.280178 (+ 5001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.282900 (+ 2722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.284858 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:35.285645 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:35.285766 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:35.304803 (+ 19037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:35.304812 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:35.304819 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:35.304862 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:35.308703 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.313689 (+ 4986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.313700 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.313703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.313923 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:35.314434 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:35.314442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:35.314447 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:35.314966 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:35.315312 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:35.316928 (+ 1616us) server_negotiation.cc:299] Negotiation successful
0620 03:04:35.317273 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21496,"thread_start_us":21608,"threads_started":1}
I20260620 03:04:35.333009 22953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b [term 1 FOLLOWER]: Refusing update from remote peer 67c809f67c2541229807729246a58e72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:04:35.334435 23004 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:04:35.336781 23023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c809f67c2541229807729246a58e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } peers { permanent_uuid: "9c626f87c3254818af032e7ca19cf222" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } } peers { permanent_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } } }
I20260620 03:04:35.337239 23023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:04:35.338208 22904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:35.338672 22878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:35.341681 22879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "67c809f67c2541229807729246a58e72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9c626f87c3254818af032e7ca19cf222"
I20260620 03:04:35.345099 22878 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222 [term 1 FOLLOWER]: Refusing update from remote peer 67c809f67c2541229807729246a58e72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:04:35.359310 22879 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67c809f67c2541229807729246a58e72 in term 1.
I20260620 03:04:35.360266 23004 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c626f87c3254818af032e7ca19cf222" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:04:35.347373 22763 master_runner.cc:186] Error getting master registration for 127.0.97.61:38173: OK, instance_id { permanent_uuid: "9c626f87c3254818af032e7ca19cf222" instance_seqno: 1781924674513343 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 03:04:35.363261 22763 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 03:04:35.381470 23023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67c809f67c2541229807729246a58e72. Latest consensus state: current_term: 1 leader_uuid: "67c809f67c2541229807729246a58e72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c809f67c2541229807729246a58e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } peers { permanent_uuid: "9c626f87c3254818af032e7ca19cf222" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } } peers { permanent_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } } }
I20260620 03:04:35.381976 23023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:04:35.385154 23022 mvcc.cc:205] Tried to move back new op lower bound from 7298763470152175616 to 7298763469599608832. Current Snapshot: MvccSnapshot[applied={T|T < 7298763470152175616}]
I20260620 03:04:35.388586 22984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67c809f67c2541229807729246a58e72. Latest consensus state: current_term: 1 leader_uuid: "67c809f67c2541229807729246a58e72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c809f67c2541229807729246a58e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } peers { permanent_uuid: "9c626f87c3254818af032e7ca19cf222" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } } peers { permanent_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } } }
I20260620 03:04:35.388981 22984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:04:35.396688 22984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67c809f67c2541229807729246a58e72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c809f67c2541229807729246a58e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } peers { permanent_uuid: "9c626f87c3254818af032e7ca19cf222" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } } peers { permanent_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } } }
I20260620 03:04:35.396960 22984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:04:35.412838 23008 catalog_manager.cc:1347] Generated new cluster ID: dce235ffb23d489b99dcaa09f8da97f7
I20260620 03:04:35.412987 23008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:04:35.414337 23028 mvcc.cc:205] Tried to move back new op lower bound from 7298763470152175616 to 7298763469599608832. Current Snapshot: MvccSnapshot[applied={T|T < 7298763470152175616}]
I20260620 03:04:35.414929 22998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67c809f67c2541229807729246a58e72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c809f67c2541229807729246a58e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } peers { permanent_uuid: "9c626f87c3254818af032e7ca19cf222" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } } peers { permanent_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } } }
I20260620 03:04:35.415328 22998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:35.415652 22998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67c809f67c2541229807729246a58e72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c809f67c2541229807729246a58e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } peers { permanent_uuid: "9c626f87c3254818af032e7ca19cf222" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } } peers { permanent_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } } }
I20260620 03:04:35.415938 22998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72 [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:35.419145 23023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67c809f67c2541229807729246a58e72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c809f67c2541229807729246a58e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } peers { permanent_uuid: "9c626f87c3254818af032e7ca19cf222" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38173 } } peers { permanent_uuid: "5d1f8eeafc734a9dbe97d6d8fa093a1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37171 } } }
I20260620 03:04:35.419420 23023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:04:35.461495 23008 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:04:35.463218 23042 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 03:04:35.463955 23008 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:04:35.478421 23008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67c809f67c2541229807729246a58e72: Generated new TSK 0
I20260620 03:04:35.479285 23008 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 03:04:35.568033 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:35.554782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49640 (local address 127.0.97.61:38173)
0620 03:04:35.555004 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.555010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.566204 (+ 11194us) server_negotiation.cc:407] Connection header received
0620 03:04:35.566421 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.566427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.566516 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.566611 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.567780 (+ 1169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49640: BlockingRecv error: recv got EOF from 127.0.0.1:49640 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 03:04:35.570343 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:35.567795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42576 (local address 127.0.97.60:37171)
0620 03:04:35.568027 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.568033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.568666 (+ 633us) server_negotiation.cc:407] Connection header received
0620 03:04:35.568911 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.568918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.568994 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.569061 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.570108 (+ 1047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42576: BlockingRecv error: recv got EOF from 127.0.0.1:42576 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 03:04:35.570569 22833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:35.555005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43994 (local address 127.0.97.62:45469)
0620 03:04:35.555231 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.555237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.567142 (+ 11905us) server_negotiation.cc:407] Connection header received
0620 03:04:35.568698 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.568705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.568799 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.568876 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.570317 (+ 1441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43994: BlockingRecv error: recv got EOF from 127.0.0.1:43994 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 03:04:35.581727 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:35.579467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49644 (local address 127.0.97.61:38173)
0620 03:04:35.579640 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.579646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.580112 (+ 466us) server_negotiation.cc:407] Connection header received
0620 03:04:35.580371 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.580377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.580507 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.580615 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.581454 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49644: BlockingRecv error: recv got EOF from 127.0.0.1:49644 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 03:04:35.589958 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:35.587370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49650 (local address 127.0.97.61:38173)
0620 03:04:35.587530 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.587537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.587982 (+ 445us) server_negotiation.cc:407] Connection header received
0620 03:04:35.588154 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.588163 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.588252 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.588361 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.589231 (+ 870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49650: BlockingRecv error: recv got EOF from 127.0.0.1:49650 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 03:04:35.605822 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:35.603480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49654 (local address 127.0.97.61:38173)
0620 03:04:35.603676 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.603682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.604190 (+ 508us) server_negotiation.cc:407] Connection header received
0620 03:04:35.604394 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.604403 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.604516 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.604634 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.605536 (+ 902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49654: BlockingRecv error: recv got EOF from 127.0.0.1:49654 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 03:04:35.615690 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:35.613168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49666 (local address 127.0.97.61:38173)
0620 03:04:35.613348 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.613354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.613872 (+ 518us) server_negotiation.cc:407] Connection header received
0620 03:04:35.614147 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.614153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.614323 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.614403 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.615453 (+ 1050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49666: BlockingRecv error: recv got EOF from 127.0.0.1:49666 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 03:04:35.629635 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:35.627274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.0.97.61:38173)
0620 03:04:35.627457 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.627463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.628016 (+ 553us) server_negotiation.cc:407] Connection header received
0620 03:04:35.628224 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.628230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.628332 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.628460 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.629346 (+ 886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49668: BlockingRecv error: recv got EOF from 127.0.0.1:49668 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 03:04:35.651558 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:35.649270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49672 (local address 127.0.97.61:38173)
0620 03:04:35.649468 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.649475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.650021 (+ 546us) server_negotiation.cc:407] Connection header received
0620 03:04:35.650231 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.650238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.650357 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.650481 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.651318 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49672: BlockingRecv error: recv got EOF from 127.0.0.1:49672 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 03:04:35.693867 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:35.691324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.0.97.61:38173)
0620 03:04:35.691522 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.691528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.692111 (+ 583us) server_negotiation.cc:407] Connection header received
0620 03:04:35.692403 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.692411 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.692545 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.692645 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.693541 (+ 896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49680: BlockingRecv error: recv got EOF from 127.0.0.1:49680 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 03:04:35.763442 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:35.761435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.0.97.61:38173)
0620 03:04:35.761692 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.761699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.762065 (+ 366us) server_negotiation.cc:407] Connection header received
0620 03:04:35.762243 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.762249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.762350 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.762489 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.763203 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49694: BlockingRecv error: recv got EOF from 127.0.0.1:49694 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 03:04:35.896689 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:35.894836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49702 (local address 127.0.97.61:38173)
0620 03:04:35.895069 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:35.895076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:35.895350 (+ 274us) server_negotiation.cc:407] Connection header received
0620 03:04:35.895574 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:35.895578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:35.895649 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:35.895750 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:35.896495 (+ 745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49702: BlockingRecv error: recv got EOF from 127.0.0.1:49702 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 03:04:36.124362 22995 catalog_manager.cc:1259] Loaded cluster ID: dce235ffb23d489b99dcaa09f8da97f7
I20260620 03:04:36.124464 22995 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b: loading cluster ID for follower catalog manager: success
I20260620 03:04:36.126917 22995 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b: acquiring CA information for follower catalog manager: success
I20260620 03:04:36.128620 22995 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5d1f8eeafc734a9dbe97d6d8fa093a1b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 03:04:36.160100 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:36.158311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49704 (local address 127.0.97.61:38173)
0620 03:04:36.158507 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:36.158515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:36.158874 (+ 359us) server_negotiation.cc:407] Connection header received
0620 03:04:36.159044 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:36.159049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:36.159129 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:36.159216 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:36.159855 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49704: BlockingRecv error: recv got EOF from 127.0.0.1:49704 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 03:04:36.467453 23042 catalog_manager.cc:1259] Loaded cluster ID: dce235ffb23d489b99dcaa09f8da97f7
I20260620 03:04:36.467555 23042 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222: loading cluster ID for follower catalog manager: success
I20260620 03:04:36.470382 23042 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222: acquiring CA information for follower catalog manager: success
I20260620 03:04:36.472178 23042 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9c626f87c3254818af032e7ca19cf222: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 03:04:36.683512 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:36.676185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.0.97.61:38173)
0620 03:04:36.677160 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:36.677167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:36.677197 (+ 30us) server_negotiation.cc:407] Connection header received
0620 03:04:36.677284 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:36.677290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:36.677481 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:36.677683 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:04:36.678064 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:36.679321 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:36.680639 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:36.681309 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:36.681450 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:36.681977 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:04:36.682841 (+ 864us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:04:36.682997 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:36.683044 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 03:04:36.683131 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":820,"thread_start_us":680,"threads_started":1}
I20260620 03:04:36.696931 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:36.690071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42586 (local address 127.0.97.60:37171)
0620 03:04:36.690621 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:36.690627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:36.690988 (+ 361us) server_negotiation.cc:407] Connection header received
0620 03:04:36.691211 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:36.691215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:36.691373 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:36.691509 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:04:36.691980 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:36.692782 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:36.694720 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:36.695109 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:36.695201 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:36.695810 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:04:36.696282 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:04:36.696389 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:36.696541 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 03:04:36.696608 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":262,"threads_started":1}
I20260620 03:04:36.724928 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:36.704583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49716 (local address 127.0.97.61:38173)
0620 03:04:36.704839 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:36.704848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:36.717823 (+ 12975us) server_negotiation.cc:407] Connection header received
0620 03:04:36.718209 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:36.718215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:36.718322 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:36.718397 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:04:36.719172 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:36.720362 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:36.722528 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:36.723238 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:36.723375 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:36.724054 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:04:36.724199 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:04:36.724265 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:36.724523 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 03:04:36.724608 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 03:04:36.728103 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:36.704546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.0.97.62:45469)
0620 03:04:36.717068 (+ 12522us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:36.717076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:36.718375 (+ 1299us) server_negotiation.cc:407] Connection header received
0620 03:04:36.718739 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:36.718745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:36.718985 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:36.719192 (+ 207us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:04:36.720262 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:36.721853 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:36.724980 (+ 3127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:36.725945 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:36.726055 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:36.726831 (+ 776us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:04:36.727349 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:04:36.727490 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:36.727656 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 03:04:36.727749 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12369,"thread_start_us":12549,"threads_started":1}
I20260620 03:04:36.732215 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:36.718600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42590 (local address 127.0.97.60:37171)
0620 03:04:36.718806 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:36.718813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:36.725844 (+ 7031us) server_negotiation.cc:407] Connection header received
0620 03:04:36.726132 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:36.726142 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:36.726257 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:36.726342 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:04:36.727174 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:36.728314 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:36.730177 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:36.730829 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:36.731012 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:36.731487 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:04:36.731628 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:04:36.731726 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:36.731847 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 03:04:36.731930 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 03:04:36.736688 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 22763
I20260620 03:04:36.737852 22825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:04:36.871407 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 22763
W20260620 03:04:36.891989 22853 connection.cc:535] server connection from 127.0.0.1:49600 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 03:04:36.892373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 22837
I20260620 03:04:36.893481 22899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:04:37.001688 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 22837
I20260620 03:04:37.022179 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 22911
I20260620 03:04:37.023304 22973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:04:37.130805 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 22911
2026-06-20T03:04:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4907 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 03:04:37 dist-test-slave-f1j0 krb5kdc[23128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 03:04:37 dist-test-slave-f1j0 krb5kdc[23128](info): set up 2 sockets
Jun 20 03:04:37 dist-test-slave-f1j0 krb5kdc[23128](info): commencing operation
krb5kdc: starting...
W20260620 03:04:38.238188 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/test-user.keytab.
Jun 20 03:04:38 dist-test-slave-f1j0 krb5kdc[23128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T03:04:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T03:04:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 03:04:38.500934 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38693
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38693,127.0.97.61:39697,127.0.97.60:46813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 03:04:38.835135 23141 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 03:04:38.835551 23141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:38.835683 23141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:38.835834 23141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:38.843168 23141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:38.843292 23141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:38.843394 23141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:38.843482 23141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:38.843570 23141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:38.853441 23141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38693,127.0.97.61:39697,127.0.97.60:46813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:38.854781 23141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:38.856076 23141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:38.867697 23146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:38.867641 23147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:38.867781 23149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:38.868499 23141 server_base.cc:1034] running on GCE node
I20260620 03:04:38.869256 23141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:38.870707 23141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:38.871954 23141 hybrid_clock.cc:648] HybridClock initialized: now 1781924678871895 us; error 40 us; skew 500 ppm
I20260620 03:04:38.872310 23141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:38 dist-test-slave-f1j0 krb5kdc[23128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:38.875582 23141 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 03:04:38.877035 23141 webserver.cc:460] Webserver started at http://127.0.97.62:34415/ using document root <none> and password file <none>
I20260620 03:04:38.877851 23141 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:38.877980 23141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:38.878314 23141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:38.881332 23141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-0/data/instance:
uuid: "7a4b5299c9da444bb2602e4368d2999e"
format_stamp: "Formatted at 2026-06-20 03:04:38 on dist-test-slave-f1j0"
server_key: "c47e13930463a0178598360d8c096295"
server_key_iv: "0cb16eada33556d14282ec29aac3e052"
server_key_version: "encryptionkey@0"
I20260620 03:04:38.882197 23141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-0/wal/instance:
uuid: "7a4b5299c9da444bb2602e4368d2999e"
format_stamp: "Formatted at 2026-06-20 03:04:38 on dist-test-slave-f1j0"
server_key: "c47e13930463a0178598360d8c096295"
server_key_iv: "0cb16eada33556d14282ec29aac3e052"
server_key_version: "encryptionkey@0"
I20260620 03:04:38.887220 23141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:04:38.890635 23156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:38.891886 23141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 03:04:38.892069 23141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-0/wal
uuid: "7a4b5299c9da444bb2602e4368d2999e"
format_stamp: "Formatted at 2026-06-20 03:04:38 on dist-test-slave-f1j0"
server_key: "c47e13930463a0178598360d8c096295"
server_key_iv: "0cb16eada33556d14282ec29aac3e052"
server_key_version: "encryptionkey@0"
I20260620 03:04:38.892295 23141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:38.914625 23141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:38.918722 23141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:38.918992 23141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:38.937619 23141 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38693
I20260620 03:04:38.937613 23207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38693 every 8 connection(s)
I20260620 03:04:38.938988 23141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-0/data/info.pb
I20260620 03:04:38.942628 23208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:38.946664 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 23141
I20260620 03:04:38.946848 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-0/wal/instance
I20260620 03:04:38.947399 388 external_mini_cluster.cc:1428] Setting key ee5439b92e498a3dafb21c27a62348bf
I20260620 03:04:38.950284 23208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:04:38 dist-test-slave-f1j0 krb5kdc[23128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260620 03:04:38.972565 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:38.954135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49158 (local address 127.0.97.62:38693)
0620 03:04:38.955783 (+ 1648us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:38.955796 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:38.955847 (+ 51us) server_negotiation.cc:407] Connection header received
0620 03:04:38.957361 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:38.957367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:38.957649 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:38.957843 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:38.959104 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:38.960281 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:38.961620 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:38.962164 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:38.962369 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:38.965519 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:38.965527 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:38.965538 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:38.965619 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:38.968390 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:38.969042 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:38.969055 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:38.969057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:38.969194 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:38.969605 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:38.969611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:38.969614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:38.970077 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:38.970287 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:38.970807 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 03:04:38.971142 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":1432,"thread_start_us":1232,"threads_started":1}
I20260620 03:04:38.972618 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:38.954133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38693 (local address 127.0.0.1:49158)
0620 03:04:38.954990 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:38.955046 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 03:04:38.957009 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:04:38.957869 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:04:38.957875 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:04:38.958570 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:04:38.958899 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:38.958916 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:38.960440 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:38.960445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:38.961470 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:38.961481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:38.962405 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:38.962412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:38.962582 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:38.963247 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:04:38.963270 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:04:38.965310 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:04:38.968509 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:38.968521 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:38.968532 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:38.968919 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:38.969312 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:38.969317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:38.969320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:38.969477 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:38.970224 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:04:38.970232 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:04:38.970439 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 03:04:38.970680 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 03:04:38.971251 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":145,"thread_start_us":382,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 03:04:38.975523 23208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:04:38.976662 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:38.976297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39697 (local address 127.0.0.1:43718)
0620 03:04:38.976447 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:38.976545 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 03:04:38.976929 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39697: connect: Connection refused (error 111)
I20260620 03:04:38.978449 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:38.978110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39697 (local address 127.0.0.1:43734)
0620 03:04:38.978256 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:38.978332 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 03:04:38.978631 23208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39697: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39697: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 03:04:39.004278 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39697
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38693,127.0.97.61:39697,127.0.97.60:46813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:04:39.027968 23208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } attempt: 1
I20260620 03:04:39.029062 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.028619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39697 (local address 127.0.0.1:43750)
0620 03:04:39.028830 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.028916 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 03:04:39.030275 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.029855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39697 (local address 127.0.0.1:43752)
0620 03:04:39.030010 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.030099 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 03:04:39.030484 23208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39697: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39697: connect: Connection refused (error 111)
I20260620 03:04:39.098752 23208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } attempt: 2
I20260620 03:04:39.100083 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.099421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39697 (local address 127.0.0.1:43768)
0620 03:04:39.099726 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.099863 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 03:04:39.101293 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.100797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39697 (local address 127.0.0.1:43784)
0620 03:04:39.100973 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.101087 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 03:04:39.101480 23208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39697: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39697: connect: Connection refused (error 111)
I20260620 03:04:39.192775 23208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } attempt: 3
I20260620 03:04:39.194262 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.193640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39697 (local address 127.0.0.1:43794)
0620 03:04:39.193896 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.194033 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 03:04:39.195274 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.194921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39697 (local address 127.0.0.1:43808)
0620 03:04:39.195060 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.195136 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 03:04:39.195523 23208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39697: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39697: connect: Connection refused (error 111)
W20260620 03:04:39.332170 23215 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 03:04:39.332479 23215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:39.332548 23215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:39.332629 23215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 03:04:39.338821 23208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } attempt: 4
W20260620 03:04:39.339862 23215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:39.339934 23215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260620 03:04:39.340023 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.339465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39697 (local address 127.0.0.1:43824)
0620 03:04:39.339755 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.339848 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260620 03:04:39.340034 23215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:39.340126 23215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:39.340262 23215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:39.341020 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.340653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39697 (local address 127.0.0.1:43838)
0620 03:04:39.340805 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.340902 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 03:04:39.341207 23208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39697: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39697: connect: Connection refused (error 111)
I20260620 03:04:39.349956 23215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38693,127.0.97.61:39697,127.0.97.60:46813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:39.351356 23215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:39.352684 23215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 03:04:39.364346 23221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:39.364226 23223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:39.364390 23220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:39.364991 23215 server_base.cc:1034] running on GCE node
I20260620 03:04:39.365878 23215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:39.367324 23215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:39.368573 23215 hybrid_clock.cc:648] HybridClock initialized: now 1781924679368499 us; error 62 us; skew 500 ppm
I20260620 03:04:39.368911 23215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:39 dist-test-slave-f1j0 krb5kdc[23128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:39.372017 23215 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 03:04:39.373378 23215 webserver.cc:460] Webserver started at http://127.0.97.61:35929/ using document root <none> and password file <none>
I20260620 03:04:39.374089 23215 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:39.374178 23215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:39.374462 23215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:39.377401 23215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-1/data/instance:
uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70"
format_stamp: "Formatted at 2026-06-20 03:04:39 on dist-test-slave-f1j0"
server_key: "144d748067ae7cee3e45b88d14a78105"
server_key_iv: "ee47c561aed731fd5b566335515f3923"
server_key_version: "encryptionkey@0"
I20260620 03:04:39.378235 23215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-1/wal/instance:
uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70"
format_stamp: "Formatted at 2026-06-20 03:04:39 on dist-test-slave-f1j0"
server_key: "144d748067ae7cee3e45b88d14a78105"
server_key_iv: "ee47c561aed731fd5b566335515f3923"
server_key_version: "encryptionkey@0"
I20260620 03:04:39.383185 23215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:04:39.386407 23230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:39.387724 23215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:04:39.387889 23215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-1/wal
uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70"
format_stamp: "Formatted at 2026-06-20 03:04:39 on dist-test-slave-f1j0"
server_key: "144d748067ae7cee3e45b88d14a78105"
server_key_iv: "ee47c561aed731fd5b566335515f3923"
server_key_version: "encryptionkey@0"
I20260620 03:04:39.388080 23215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:39.423435 23215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:39.427727 23215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:39.428006 23215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:39.446374 23215 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39697
I20260620 03:04:39.446398 23281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39697 every 8 connection(s)
I20260620 03:04:39.447763 23215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-1/data/info.pb
I20260620 03:04:39.448864 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 23215
I20260620 03:04:39.449049 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-1/wal/instance
I20260620 03:04:39.449489 388 external_mini_cluster.cc:1428] Setting key 3e675eaa4d8456c4146f92a73e8dab2f
I20260620 03:04:39.451637 23282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:39.458305 23282 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:04:39 dist-test-slave-f1j0 krb5kdc[23128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260620 03:04:39.480492 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.461306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49168 (local address 127.0.97.62:38693)
0620 03:04:39.461617 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:39.461625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:39.462981 (+ 1356us) server_negotiation.cc:407] Connection header received
0620 03:04:39.465153 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:39.465161 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:39.465303 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:39.465437 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:39.467668 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.468859 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.470657 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.471065 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.471203 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:39.474685 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:39.474690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:39.474693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:39.474726 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:39.477332 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:39.478152 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:39.478162 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:39.478166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:39.478290 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:39.478736 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:39.478740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:39.478742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:39.479034 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:39.479151 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:39.480097 (+ 946us) server_negotiation.cc:299] Negotiation successful
0620 03:04:39.480269 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260620 03:04:39.482115 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.461800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38693 (local address 127.0.0.1:49168)
0620 03:04:39.462819 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.462856 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 03:04:39.464731 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:04:39.465725 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:04:39.465737 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:04:39.466586 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:04:39.467442 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:39.467479 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.469077 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.469086 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:39.470503 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:39.470519 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.471291 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.471299 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:39.471549 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:39.472303 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:04:39.472325 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:04:39.474455 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:04:39.477466 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:39.477475 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:39.477491 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:39.478026 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:39.478399 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:39.478405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:39.478408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:39.478590 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:39.479175 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:04:39.479186 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:04:39.479671 (+ 485us) client_negotiation.cc:769] Sending connection context
0620 03:04:39.480005 (+ 334us) client_negotiation.cc:240] Negotiation successful
0620 03:04:39.480575 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":712,"threads_started":1}
I20260620 03:04:39.483875 23282 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:04:39 dist-test-slave-f1j0 krb5kdc[23128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:04:39.500484 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.484641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39697 (local address 127.0.0.1:43850)
0620 03:04:39.484854 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.484876 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:04:39.484983 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:04:39.485988 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:04:39.485996 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:04:39.486514 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:04:39.486733 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:39.486746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.488122 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.488129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:39.489404 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:39.489416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.490665 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.490669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:39.490805 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:39.491422 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:04:39.491439 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:04:39.493644 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:04:39.498016 (+ 4372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:39.498021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:39.498024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:39.498413 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:39.498849 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:39.498853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:39.498855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:39.498991 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:39.499767 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:04:39.499771 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:04:39.499907 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 03:04:39.500019 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 03:04:39.500219 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 03:04:39.501013 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.484847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43850 (local address 127.0.97.61:39697)
0620 03:04:39.485506 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:39.485512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:39.485536 (+ 24us) server_negotiation.cc:407] Connection header received
0620 03:04:39.485632 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:39.485639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:39.485835 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:39.485951 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:39.486910 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.487962 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.489537 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.490531 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.490675 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:39.493871 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:39.493885 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:39.493905 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:39.493946 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:39.497851 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:39.498552 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:39.498564 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:39.498571 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:39.498720 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:39.499089 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:39.499096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:39.499102 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:39.499607 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:39.499819 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:39.500198 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 03:04:39.500497 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":383,"threads_started":1}
I20260620 03:04:39.503584 23282 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:04:39.504693 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.504262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38752)
0620 03:04:39.504426 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.504537 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 03:04:39.504997 23231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
I20260620 03:04:39.506556 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.506214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38766)
0620 03:04:39.506381 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.506458 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 03:04:39.506808 23282 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46813: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 03:04:39.514678 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRaC98/build/asan/bin/kudu
/tmp/dist-test-taskpRaC98/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46813
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38693,127.0.97.61:39697,127.0.97.60:46813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 03:04:39.556140 23282 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } attempt: 1
I20260620 03:04:39.557369 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.556917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38778)
0620 03:04:39.557123 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.557215 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 03:04:39.558405 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.558037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38784)
0620 03:04:39.558187 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.558291 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 03:04:39.558691 23282 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46813: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
I20260620 03:04:39.626937 23282 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } attempt: 2
I20260620 03:04:39.628055 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.627598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38798)
0620 03:04:39.627803 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.627899 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 03:04:39.629024 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.628687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38804)
0620 03:04:39.628849 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.628913 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 03:04:39.629235 23282 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46813: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
I20260620 03:04:39.640467 23208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } attempt: 5
Jun 20 03:04:39 dist-test-slave-f1j0 krb5kdc[23128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:04:39.655145 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.641191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39697 (local address 127.0.0.1:43852)
0620 03:04:39.641402 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.641424 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 03:04:39.641611 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:04:39.641982 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:04:39.641987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:04:39.642530 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:04:39.642783 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:39.642796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.643761 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.643767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:39.645238 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:39.645253 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.646083 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.646092 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:39.646397 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:39.647117 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:04:39.647148 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:04:39.649757 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:04:39.652660 (+ 2903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:39.652666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:39.652670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:39.653048 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:39.653422 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:39.653427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:39.653431 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:39.653563 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:39.654115 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:04:39.654120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:04:39.654335 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 03:04:39.654480 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 03:04:39.654788 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 03:04:39.655149 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.641287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43852 (local address 127.0.97.61:39697)
0620 03:04:39.641469 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:39.641474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:39.641492 (+ 18us) server_negotiation.cc:407] Connection header received
0620 03:04:39.641738 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:39.641742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:39.641831 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:39.641916 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:39.642926 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.643603 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.645388 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.645914 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.646012 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:39.649949 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:39.649954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:39.649957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:39.649988 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:39.652521 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:39.653165 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:39.653172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:39.653176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:39.653304 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:39.653697 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:39.653703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:39.653706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:39.653994 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:39.654153 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:39.654599 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 03:04:39.654809 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 03:04:39.656185 23208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:04:39.657287 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.656915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38818)
0620 03:04:39.657081 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.657150 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 03:04:39.658236 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.657922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38822)
0620 03:04:39.658063 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.658130 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 03:04:39.658449 23208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46813: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
I20260620 03:04:39.709697 23208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } attempt: 1
I20260620 03:04:39.710824 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.710365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38826)
0620 03:04:39.710561 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.710658 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 03:04:39.711880 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.711566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38828)
0620 03:04:39.711710 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.711777 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 03:04:39.712086 23208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46813: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
I20260620 03:04:39.720534 23282 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } attempt: 3
I20260620 03:04:39.721527 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.721115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38842)
0620 03:04:39.721306 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.721389 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 03:04:39.722815 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.722337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38858)
0620 03:04:39.722600 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.722699 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
W20260620 03:04:39.723129 23282 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46813: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
I20260620 03:04:39.780306 23208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } attempt: 2
I20260620 03:04:39.781445 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.780963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38862)
0620 03:04:39.781161 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.781278 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 03:04:39.782473 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.782156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38872)
0620 03:04:39.782310 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.782374 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 03:04:39.782684 23208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46813: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
W20260620 03:04:39.845706 23289 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 03:04:39.846082 23289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 03:04:39.846201 23289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 03:04:39.846329 23289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 03:04:39.853561 23289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 03:04:39.853734 23289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 03:04:39.853845 23289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 03:04:39.853938 23289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 03:04:39.854024 23289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 03:04:39.864038 23289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38693,127.0.97.61:39697,127.0.97.60:46813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 03:04:39.865347 23289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 03:04:39.866341 23282 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } attempt: 4
I20260620 03:04:39.866788 23289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 03:04:39.867427 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.866987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38886)
0620 03:04:39.867196 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.867285 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 03:04:39.868417 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.868123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38890)
0620 03:04:39.868250 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.868317 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 03:04:39.868678 23282 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46813: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
W20260620 03:04:39.878888 23294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:39.878957 23297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 03:04:39.878928 23295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 03:04:39.879859 23289 server_base.cc:1034] running on GCE node
I20260620 03:04:39.880705 23289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 03:04:39.882239 23289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 03:04:39.883481 23289 hybrid_clock.cc:648] HybridClock initialized: now 1781924679883397 us; error 58 us; skew 500 ppm
I20260620 03:04:39.883813 23289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 03:04:39 dist-test-slave-f1j0 krb5kdc[23128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 03:04:39.886977 23289 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 03:04:39.888458 23289 webserver.cc:460] Webserver started at http://127.0.97.60:44813/ using document root <none> and password file <none>
I20260620 03:04:39.888870 23208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } attempt: 3
I20260620 03:04:39.889204 23289 fs_manager.cc:359] Metadata directory not provided
I20260620 03:04:39.889322 23289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 03:04:39.889684 23289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 03:04:39.890002 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.889468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38892)
0620 03:04:39.889719 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.889841 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 03:04:39.891029 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.890656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38898)
0620 03:04:39.890860 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.890930 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260620 03:04:39.891227 23208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46813: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46813: connect: Connection refused (error 111)
I20260620 03:04:39.892997 23289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-2/data/instance:
uuid: "3c27734695f94408b89668adaabf0126"
format_stamp: "Formatted at 2026-06-20 03:04:39 on dist-test-slave-f1j0"
server_key: "5518a9292a8d4fbad0202a544bcbfaa7"
server_key_iv: "1f15ffd31decfb3b6618eff162488ce1"
server_key_version: "encryptionkey@0"
I20260620 03:04:39.893822 23289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-2/wal/instance:
uuid: "3c27734695f94408b89668adaabf0126"
format_stamp: "Formatted at 2026-06-20 03:04:39 on dist-test-slave-f1j0"
server_key: "5518a9292a8d4fbad0202a544bcbfaa7"
server_key_iv: "1f15ffd31decfb3b6618eff162488ce1"
server_key_version: "encryptionkey@0"
I20260620 03:04:39.898806 23289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 03:04:39.902153 23304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 03:04:39.903442 23289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 03:04:39.903635 23289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-2/wal
uuid: "3c27734695f94408b89668adaabf0126"
format_stamp: "Formatted at 2026-06-20 03:04:39 on dist-test-slave-f1j0"
server_key: "5518a9292a8d4fbad0202a544bcbfaa7"
server_key_iv: "1f15ffd31decfb3b6618eff162488ce1"
server_key_version: "encryptionkey@0"
I20260620 03:04:39.903846 23289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 03:04:39.930101 23289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 03:04:39.934387 23289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 03:04:39.934659 23289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 03:04:39.953503 23289 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46813
I20260620 03:04:39.953518 23355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46813 every 8 connection(s)
I20260620 03:04:39.954880 23289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-2/data/info.pb
I20260620 03:04:39.958503 23356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 03:04:39.959200 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRaC98/build/asan/bin/kudu as pid 23289
I20260620 03:04:39.959388 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRaC98/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781924320356866-388-0/minicluster-data/master-2/wal/instance
I20260620 03:04:39.965185 388 external_mini_cluster.cc:1428] Setting key 7f32830300a76590fa0a007e61e1d08d
I20260620 03:04:39.968230 23356 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:04:39 dist-test-slave-f1j0 krb5kdc[23128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 03:04:39 dist-test-slave-f1j0 krb5kdc[23128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 03:04:39.986090 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.970830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49174 (local address 127.0.97.62:38693)
0620 03:04:39.971065 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:39.971073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:39.971762 (+ 689us) server_negotiation.cc:407] Connection header received
0620 03:04:39.972064 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:39.972071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:39.972162 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:39.972279 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:39.973670 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.974417 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.975945 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.976386 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.976515 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:39.979989 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:39.979998 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:39.980003 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:39.980047 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:39.982744 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:39.983560 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:39.983567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:39.983571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:39.983733 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:39.984117 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:39.984123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:39.984126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:39.984474 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:39.984627 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:39.985418 (+ 791us) server_negotiation.cc:299] Negotiation successful
0620 03:04:39.985687 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 03:04:39.987208 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.967191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38910 (local address 127.0.97.60:46813)
0620 03:04:39.968022 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:39.968034 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:39.968077 (+ 43us) server_negotiation.cc:407] Connection header received
0620 03:04:39.970529 (+ 2452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:39.970552 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:39.971100 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:39.971567 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:39.973082 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.974549 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.975756 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.976323 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.976465 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:39.980657 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:39.980669 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:39.980685 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:39.980727 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:39.983647 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:39.984364 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:39.984370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:39.984372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:39.984477 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:39.984842 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:39.984847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:39.984850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:39.985245 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:39.985509 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:39.985843 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 03:04:39.986060 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":578,"spinlock_wait_cycles":15232,"thread_start_us":356,"threads_started":1}
I20260620 03:04:39.987246 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.970816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38693 (local address 127.0.0.1:49174)
0620 03:04:39.971623 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.971673 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 03:04:39.971910 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:04:39.972334 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:04:39.972339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:04:39.972897 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:04:39.973470 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:39.973493 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.974567 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.974574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:39.975806 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:39.975825 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.976621 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.976628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:39.976768 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:39.977444 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:04:39.977469 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:04:39.979790 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:04:39.982894 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:39.982905 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:39.982917 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:39.983437 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:39.983838 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:39.983841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:39.983844 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:39.984009 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:39.984596 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:04:39.984603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:04:39.984995 (+ 392us) client_negotiation.cc:769] Sending connection context
0620 03:04:39.985312 (+ 317us) client_negotiation.cc:240] Negotiation successful
0620 03:04:39.985848 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"spinlock_wait_cycles":22656,"thread_start_us":380,"threads_started":1}
I20260620 03:04:39.988761 23356 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 03:04:39 dist-test-slave-f1j0 krb5kdc[23128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 03:04:39 dist-test-slave-f1j0 krb5kdc[23128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 03:04:40.003494 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.989423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39697 (local address 127.0.0.1:43864)
0620 03:04:39.989638 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:39.989656 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 03:04:39.989742 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:04:39.990098 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:04:39.990104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:04:39.990510 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:04:39.990707 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:39.990721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.992118 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.992122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:39.993485 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:39.993501 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.994416 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.994420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:39.994559 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:39.995127 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:04:39.995151 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:04:39.997834 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:04:40.000784 (+ 2950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:40.000803 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:40.000808 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:40.001307 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:40.001727 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:40.001731 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:40.001733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:40.001927 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:40.002446 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:04:40.002451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:04:40.002608 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 03:04:40.002759 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 03:04:40.003093 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 03:04:40.003572 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.989518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43864 (local address 127.0.97.61:39697)
0620 03:04:39.989750 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:39.989756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:39.989798 (+ 42us) server_negotiation.cc:407] Connection header received
0620 03:04:39.989864 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:39.989872 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:39.989954 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:39.990058 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:39.990847 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.991973 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.993640 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.994288 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.994416 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:39.998010 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:39.998015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:39.998018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:39.998050 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:40.000655 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:40.001423 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:40.001427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:40.001429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:40.001560 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:40.002043 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:40.002047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:40.002049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:40.002315 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:40.002449 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:40.003031 (+ 582us) server_negotiation.cc:299] Negotiation successful
0620 03:04:40.003250 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 03:04:40.004683 23356 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } has no permanent_uuid. Determining permanent_uuid...
I20260620 03:04:40.006714 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:39.990464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.0.97.62:38693)
0620 03:04:39.990621 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:39.990628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:39.990650 (+ 22us) server_negotiation.cc:407] Connection header received
0620 03:04:39.990769 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:39.990777 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:39.990855 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:39.990930 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:39.992358 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.993428 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.995081 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:39.995835 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:39.995955 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:40.000085 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:40.000091 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:40.000100 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:40.000130 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:40.003378 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:40.004425 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:40.004434 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:40.004441 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:40.004611 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:40.005075 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:40.005080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:40.005083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:40.005400 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:40.005705 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:40.006097 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 03:04:40.006355 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 03:04:40 dist-test-slave-f1j0 krb5kdc[23128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 03:04:40 dist-test-slave-f1j0 krb5kdc[23128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 03:04:40.019093 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:40.005567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38920)
0620 03:04:40.005802 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:40.005818 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 03:04:40.005919 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:04:40.006245 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:04:40.006249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:04:40.006744 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:04:40.006967 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:40.006977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.008089 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.008097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:40.009502 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:40.009521 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.010352 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.010358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:40.010562 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:40.011202 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:04:40.011224 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:04:40.013685 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:04:40.016766 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:40.016772 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:40.016779 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:40.017186 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:40.017558 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:40.017564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:40.017592 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:40.017718 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:40.018209 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:04:40.018213 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:04:40.018371 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 03:04:40.018518 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 03:04:40.018810 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 03:04:40.019426 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:40.005855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38920 (local address 127.0.97.60:46813)
0620 03:04:40.006024 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.006029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.006043 (+ 14us) server_negotiation.cc:407] Connection header received
0620 03:04:40.006082 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.006085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.006134 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.006230 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:40.007110 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.007977 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.009699 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.010191 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.010302 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:40.013902 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:40.013907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:40.013911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:40.013939 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:40.016619 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:40.017325 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:40.017330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:40.017334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:40.017455 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:40.017828 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:40.017832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:40.017836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:40.018098 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:40.018248 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:40.018654 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 03:04:40.019015 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":56}
I20260620 03:04:40.023967 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:40.008477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43866 (local address 127.0.97.61:39697)
0620 03:04:40.008635 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.008639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.008655 (+ 16us) server_negotiation.cc:407] Connection header received
0620 03:04:40.008780 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.008783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.008843 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.008922 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:40.010430 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.011383 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.013030 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.013522 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.013709 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:40.017798 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:40.017804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:40.017807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:40.017843 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:40.021138 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:40.021880 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:40.021887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:40.021891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:40.022041 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:40.022409 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:40.022414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:40.022418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:40.022751 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:40.023014 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:40.023395 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 03:04:40.023626 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:04:40.025414 23356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126: Bootstrap starting.
I20260620 03:04:40.028628 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 03:04:40.028941 23356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:40.030030 23356 log.cc:824] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:40.032786 23356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126: No bootstrap required, opened a new log
I20260620 03:04:40.031001 23183 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7a4b5299c9da444bb2602e4368d2999e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49178
I20260620 03:04:40.039788 23356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4b5299c9da444bb2602e4368d2999e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } } peers { permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } } peers { permanent_uuid: "3c27734695f94408b89668adaabf0126" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } }
I20260620 03:04:40.040112 23356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c27734695f94408b89668adaabf0126, State: Initialized, Role: FOLLOWER
I20260620 03:04:40.040568 23356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4b5299c9da444bb2602e4368d2999e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } } peers { permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } } peers { permanent_uuid: "3c27734695f94408b89668adaabf0126" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } }
I20260620 03:04:40.041463 23362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4b5299c9da444bb2602e4368d2999e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } } peers { permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } } peers { permanent_uuid: "3c27734695f94408b89668adaabf0126" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } } }
I20260620 03:04:40.041899 23362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:04:40.042460 23356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 03:04:40.052404 23373 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 03:04:40.052560 23373 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 03:04:40.049965 23289 master_runner.cc:186] Error getting master registration for 127.0.97.62:38693: OK, instance_id { permanent_uuid: "7a4b5299c9da444bb2602e4368d2999e" instance_seqno: 1781924678932376 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 03:04:40.061625 23289 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 03:04:40.068435 23208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } attempt: 4
Jun 20 03:04:40 dist-test-slave-f1j0 krb5kdc[23128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 03:04:40.085819 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:40.069462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.0.97.60:46813)
0620 03:04:40.069703 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.069709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.069732 (+ 23us) server_negotiation.cc:407] Connection header received
0620 03:04:40.069800 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.069805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.069888 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.070005 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:40.070975 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.072162 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.074026 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.074687 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.074846 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:40.079062 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:40.079070 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:40.079076 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:40.079115 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:40.082724 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:40.083552 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:40.083558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:40.083561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:40.083712 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:40.084141 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:40.084146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:40.084149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:40.084608 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:40.084754 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:40.085182 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 03:04:40.085428 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 03:04:40.085824 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:40.069198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38932)
0620 03:04:40.069418 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:40.069441 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 03:04:40.069609 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:04:40.070056 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:04:40.070062 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:04:40.070587 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:04:40.070806 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:40.070823 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.072318 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.072323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:40.073892 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:40.073912 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.074863 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.074869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:40.075098 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:40.075790 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:04:40.075815 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:04:40.078854 (+ 3039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:04:40.082868 (+ 4014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:40.082873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:40.082878 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:40.083433 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:40.083833 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:40.083838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:40.083842 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:40.084024 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:40.084733 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:04:40.084739 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:04:40.084914 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 03:04:40.085064 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 03:04:40.085380 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 03:04:40.090371 23208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e: Bootstrap starting.
I20260620 03:04:40.094511 23208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:40.095825 23208 log.cc:824] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:40.098968 23208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e: No bootstrap required, opened a new log
I20260620 03:04:40.106806 23208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4b5299c9da444bb2602e4368d2999e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } } peers { permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } } peers { permanent_uuid: "3c27734695f94408b89668adaabf0126" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } }
I20260620 03:04:40.107244 23208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a4b5299c9da444bb2602e4368d2999e, State: Initialized, Role: FOLLOWER
I20260620 03:04:40.107720 23208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4b5299c9da444bb2602e4368d2999e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } } peers { permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } } peers { permanent_uuid: "3c27734695f94408b89668adaabf0126" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } }
I20260620 03:04:40.108734 23376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4b5299c9da444bb2602e4368d2999e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } } peers { permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } } peers { permanent_uuid: "3c27734695f94408b89668adaabf0126" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } } }
I20260620 03:04:40.109169 23376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:04:40.109980 23208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:40.110122 23183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 03:04:40.110348 23183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:40.113981 23183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4b5299c9da444bb2602e4368d2999e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } } peers { permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } } peers { permanent_uuid: "3c27734695f94408b89668adaabf0126" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } }
I20260620 03:04:40.115619 23183 leader_election.cc:288] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [CANDIDATE]: Term 1 election: Requested vote from peers fcf1fd7c2f8344c8acccd5af9cc67c70 (127.0.97.61:39697), 3c27734695f94408b89668adaabf0126 (127.0.97.60:46813)
I20260620 03:04:40.118602 23331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a4b5299c9da444bb2602e4368d2999e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3c27734695f94408b89668adaabf0126"
I20260620 03:04:40.119707 23331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:40.124797 23331 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a4b5299c9da444bb2602e4368d2999e in term 1.
I20260620 03:04:40.125788 23157 leader_election.cc:302] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c27734695f94408b89668adaabf0126, 7a4b5299c9da444bb2602e4368d2999e; no voters:
I20260620 03:04:40.126415 23376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 03:04:40.128584 23376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [term 1 LEADER]: Becoming Leader. State: Replica: 7a4b5299c9da444bb2602e4368d2999e, State: Running, Role: LEADER
I20260620 03:04:40.129254 23376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4b5299c9da444bb2602e4368d2999e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } } peers { permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } } peers { permanent_uuid: "3c27734695f94408b89668adaabf0126" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } }
I20260620 03:04:40.136780 23384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a4b5299c9da444bb2602e4368d2999e. Latest consensus state: current_term: 1 leader_uuid: "7a4b5299c9da444bb2602e4368d2999e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4b5299c9da444bb2602e4368d2999e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } } peers { permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } } peers { permanent_uuid: "3c27734695f94408b89668adaabf0126" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } } }
I20260620 03:04:40.137346 23384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:40.138594 23389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 03:04:40.156242 23389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 03:04:40.167834 23282 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } attempt: 5
W20260620 03:04:40.171878 23162 tablet.cc:2369] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:04:40.173978 23331 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126 [term 1 FOLLOWER]: Refusing update from remote peer 7a4b5299c9da444bb2602e4368d2999e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:04:40.176323 23384 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c27734695f94408b89668adaabf0126" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:04:40.178639 23400 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260620 03:04:40.186621 23310 tablet.cc:2369] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 03:04:40.188179 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:40.151662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43870 (local address 127.0.97.61:39697)
0620 03:04:40.151840 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.151846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.158250 (+ 6404us) server_negotiation.cc:407] Connection header received
0620 03:04:40.158598 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.158606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.158725 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.158835 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:40.170755 (+ 11920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.171876 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.174855 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.175554 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.175703 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:40.180028 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:40.180035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:40.180040 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:40.180081 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:40.184240 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:40.185381 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:40.185388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:40.185393 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:40.185598 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:40.186215 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:40.186221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:40.186225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:40.186605 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:40.186978 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:40.187552 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 03:04:40.187785 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 03:04:40.197367 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:40.151248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49184 (local address 127.0.97.62:38693)
0620 03:04:40.151440 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.151448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.162610 (+ 11162us) server_negotiation.cc:407] Connection header received
0620 03:04:40.162776 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.162782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.162864 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.162958 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:40.173534 (+ 10576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.174777 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.176698 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.177437 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.177541 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:40.182847 (+ 5306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:40.182856 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:40.182863 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:40.182911 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:40.187016 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:40.193695 (+ 6679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:40.193701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:40.193706 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:40.193942 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:40.194523 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:40.194532 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:40.194537 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:40.194989 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:40.195347 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:40.196545 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0620 03:04:40.196862 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 03:04:40.184001 23141 master_runner.cc:186] Error getting master registration for 127.0.97.61:39697: OK, instance_id { permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" instance_seqno: 1781924679441129 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
Jun 20 03:04:40 dist-test-slave-f1j0 krb5kdc[23128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 03:04:40.210629 23141 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 03:04:40.218412 23362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a4b5299c9da444bb2602e4368d2999e. Latest consensus state: current_term: 1 leader_uuid: "7a4b5299c9da444bb2602e4368d2999e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4b5299c9da444bb2602e4368d2999e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } } peers { permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } } peers { permanent_uuid: "3c27734695f94408b89668adaabf0126" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } } }
I20260620 03:04:40.218791 23362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:04:40.218729 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:40.161818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38944 (local address 127.0.97.60:46813)
0620 03:04:40.162042 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.162051 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.163557 (+ 1506us) server_negotiation.cc:407] Connection header received
0620 03:04:40.170619 (+ 7062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.170628 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.170771 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.170858 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:40.173445 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.174682 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.179237 (+ 4555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.180063 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.180157 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:40.209874 (+ 29717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:40.209882 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:40.209889 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:40.209934 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:40.214398 (+ 4464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:40.215417 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:40.215424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:40.215430 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:40.215574 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:40.216096 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:40.216101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:40.216104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:40.216546 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:40.216976 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:40.217933 (+ 957us) server_negotiation.cc:299] Negotiation successful
0620 03:04:40.218280 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 03:04:40.228931 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:40.168457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46813 (local address 127.0.0.1:38950)
0620 03:04:40.168862 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 03:04:40.168881 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 03:04:40.168983 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 03:04:40.193904 (+ 24921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 03:04:40.193912 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 03:04:40.194389 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 03:04:40.194611 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:40.194626 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.198134 (+ 3508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.198140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:40.199546 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 03:04:40.199562 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.201210 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.201217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 03:04:40.201427 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:40.202132 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 03:04:40.202169 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 03:04:40.205138 (+ 2969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 03:04:40.218002 (+ 12864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:40.218007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:40.218011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:40.218703 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:40.219441 (+ 738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 03:04:40.219446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 03:04:40.219450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 03:04:40.219653 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 03:04:40.227451 (+ 7798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 03:04:40.227460 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 03:04:40.227710 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 03:04:40.227888 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 03:04:40.228321 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287}
I20260620 03:04:40.231536 23362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a4b5299c9da444bb2602e4368d2999e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4b5299c9da444bb2602e4368d2999e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } } peers { permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } } peers { permanent_uuid: "3c27734695f94408b89668adaabf0126" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } } }
I20260620 03:04:40.231884 23362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:04:40.236331 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:40.169272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38950 (local address 127.0.97.60:46813)
0620 03:04:40.193321 (+ 24049us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.193328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.193374 (+ 46us) server_negotiation.cc:407] Connection header received
0620 03:04:40.193457 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.193484 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.193667 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.194822 (+ 1155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:40.195533 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.197897 (+ 2364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.199816 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.200943 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.201874 (+ 931us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:40.205631 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 03:04:40.205666 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 03:04:40.205673 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 03:04:40.205740 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 03:04:40.217787 (+ 12047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:40.218919 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:40.218955 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:40.218961 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:40.219256 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 03:04:40.225033 (+ 5777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 03:04:40.225045 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 03:04:40.225051 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 03:04:40.227023 (+ 1972us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 03:04:40.230052 (+ 3029us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:40.230869 (+ 817us) server_negotiation.cc:299] Negotiation successful
0620 03:04:40.235104 (+ 4235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23819,"thread_start_us":23388,"threads_started":1}
I20260620 03:04:40.242748 23384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a4b5299c9da444bb2602e4368d2999e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4b5299c9da444bb2602e4368d2999e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } } peers { permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } } peers { permanent_uuid: "3c27734695f94408b89668adaabf0126" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } } }
I20260620 03:04:40.243038 23384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:40.243528 23282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70: Bootstrap starting.
I20260620 03:04:40.248659 23282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70: Neither blocks nor log segments found. Creating new log.
I20260620 03:04:40.249764 23389 catalog_manager.cc:1347] Generated new cluster ID: feab58c59b3440e5aecf3865ac62a7c4
I20260620 03:04:40.249866 23389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 03:04:40.250010 23282 log.cc:824] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70: Log is configured to *not* fsync() on all Append() calls
I20260620 03:04:40.256256 23282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70: No bootstrap required, opened a new log
I20260620 03:04:40.268959 23282 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4b5299c9da444bb2602e4368d2999e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } } peers { permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } } peers { permanent_uuid: "3c27734695f94408b89668adaabf0126" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } }
I20260620 03:04:40.269471 23282 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcf1fd7c2f8344c8acccd5af9cc67c70, State: Initialized, Role: FOLLOWER
I20260620 03:04:40.270080 23282 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4b5299c9da444bb2602e4368d2999e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } } peers { permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } } peers { permanent_uuid: "3c27734695f94408b89668adaabf0126" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } }
I20260620 03:04:40.280920 23406 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4b5299c9da444bb2602e4368d2999e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } } peers { permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } } peers { permanent_uuid: "3c27734695f94408b89668adaabf0126" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } } }
I20260620 03:04:40.283356 23406 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:04:40.283221 23282 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 03:04:40.283778 23256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70 [term 0 FOLLOWER]: Advancing to term 1
I20260620 03:04:40.284786 23389 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 03:04:40.287009 23389 catalog_manager.cc:1504] Loading token signing keys...
I20260620 03:04:40.284297 23257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a4b5299c9da444bb2602e4368d2999e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70"
I20260620 03:04:40.295171 23256 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70 [term 1 FOLLOWER]: Refusing update from remote peer 7a4b5299c9da444bb2602e4368d2999e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 03:04:40.300010 23257 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a4b5299c9da444bb2602e4368d2999e in term 1.
I20260620 03:04:40.309806 23384 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 03:04:40.313467 23389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e: Generated new TSK 0
I20260620 03:04:40.314500 23389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 03:04:40.356711 23406 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a4b5299c9da444bb2602e4368d2999e. Latest consensus state: current_term: 1 leader_uuid: "7a4b5299c9da444bb2602e4368d2999e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4b5299c9da444bb2602e4368d2999e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } } peers { permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } } peers { permanent_uuid: "3c27734695f94408b89668adaabf0126" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } } }
I20260620 03:04:40.357147 23406 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:04:40.357846 23413 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a4b5299c9da444bb2602e4368d2999e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4b5299c9da444bb2602e4368d2999e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } } peers { permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } } peers { permanent_uuid: "3c27734695f94408b89668adaabf0126" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } } }
I20260620 03:04:40.358410 23413 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 03:04:40.359975 23376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a4b5299c9da444bb2602e4368d2999e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4b5299c9da444bb2602e4368d2999e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38693 } } peers { permanent_uuid: "fcf1fd7c2f8344c8acccd5af9cc67c70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39697 } } peers { permanent_uuid: "3c27734695f94408b89668adaabf0126" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46813 } } }
I20260620 03:04:40.360389 23376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4b5299c9da444bb2602e4368d2999e [sys.catalog]: This master's current role is: LEADER
I20260620 03:04:40.399736 23423 catalog_manager.cc:1259] Loaded cluster ID: feab58c59b3440e5aecf3865ac62a7c4
I20260620 03:04:40.399856 23423 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70: loading cluster ID for follower catalog manager: success
I20260620 03:04:40.411350 23423 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70: acquiring CA information for follower catalog manager: success
I20260620 03:04:40.422147 23423 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fcf1fd7c2f8344c8acccd5af9cc67c70: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 03:04:40.500733 23211 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:40.486519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49186 (local address 127.0.97.62:38693)
0620 03:04:40.487133 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.487149 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.498241 (+ 11092us) server_negotiation.cc:407] Connection header received
0620 03:04:40.498489 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.498495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.498600 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.498749 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:40.500489 (+ 1740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49186: BlockingRecv error: recv got EOF from 127.0.0.1:49186 (error 108)
Metrics: {"server-negotiator.queue_time_us":467}
W20260620 03:04:40.500727 23286 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:40.486703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43874 (local address 127.0.97.61:39697)
0620 03:04:40.486875 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.486881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.498887 (+ 12006us) server_negotiation.cc:407] Connection header received
0620 03:04:40.499045 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.499050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.499167 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.499277 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:40.500471 (+ 1194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43874: BlockingRecv error: recv got EOF from 127.0.0.1:43874 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 03:04:40.501405 23398 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:40.486908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38962 (local address 127.0.97.60:46813)
0620 03:04:40.487096 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.487102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.499478 (+ 12376us) server_negotiation.cc:407] Connection header received
0620 03:04:40.499739 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.499746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.499863 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.499934 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:40.501174 (+ 1240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38962: BlockingRecv error: recv got EOF from 127.0.0.1:38962 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 03:04:40.558207 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:40.538671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43890 (local address 127.0.97.61:39697)
0620 03:04:40.538855 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.538862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.550176 (+ 11314us) server_negotiation.cc:407] Connection header received
0620 03:04:40.550540 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.550546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.550794 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.550942 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:04:40.551659 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.552874 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.554988 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.555675 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.555779 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:40.556544 (+ 765us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:04:40.557324 (+ 780us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:04:40.557441 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:40.557739 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 03:04:40.557842 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 03:04:40.604871 23398 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:40.591596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38970 (local address 127.0.97.60:46813)
0620 03:04:40.591809 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.591816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.602630 (+ 10814us) server_negotiation.cc:407] Connection header received
0620 03:04:40.603202 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.603209 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.603367 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.603452 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:40.604573 (+ 1121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38970: BlockingRecv error: recv got EOF from 127.0.0.1:38970 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 03:04:40.669469 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:40.644200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.0.97.61:39697)
0620 03:04:40.644460 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.644470 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.662250 (+ 17780us) server_negotiation.cc:407] Connection header received
0620 03:04:40.662495 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.662502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.662622 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.662729 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:04:40.663379 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.664573 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.667152 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.667932 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.668080 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:40.668597 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:04:40.668768 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:04:40.668873 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:40.669029 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 03:04:40.669120 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260620 03:04:40.681463 23398 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:40.676930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38976 (local address 127.0.97.60:46813)
0620 03:04:40.677123 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.677132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.679912 (+ 2780us) server_negotiation.cc:407] Connection header received
0620 03:04:40.680115 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.680121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.680224 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.680338 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:40.681221 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38976: BlockingRecv error: recv got EOF from 127.0.0.1:38976 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 03:04:40.698051 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:40.690611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43904 (local address 127.0.97.61:39697)
0620 03:04:40.690818 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.690826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.691395 (+ 569us) server_negotiation.cc:407] Connection header received
0620 03:04:40.691577 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.691583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.691714 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.691848 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:04:40.692531 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.693854 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.695855 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.696557 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.696685 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:40.697167 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:04:40.697316 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:04:40.697422 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:40.697604 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 03:04:40.697695 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 03:04:40.707338 23398 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:40.704999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38980 (local address 127.0.97.60:46813)
0620 03:04:40.705194 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.705200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.705730 (+ 530us) server_negotiation.cc:407] Connection header received
0620 03:04:40.705956 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.705962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.706082 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.706210 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:40.707074 (+ 864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38980: BlockingRecv error: recv got EOF from 127.0.0.1:38980 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 03:04:40.726873 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:40.719742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43912 (local address 127.0.97.61:39697)
0620 03:04:40.719968 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.719975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.720496 (+ 521us) server_negotiation.cc:407] Connection header received
0620 03:04:40.720706 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.720712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.720819 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.720934 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:04:40.721410 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.722655 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.724636 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.725381 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.725521 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:40.726062 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:04:40.726217 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:04:40.726327 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:40.726497 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 03:04:40.726585 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260620 03:04:40.742307 23398 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:40.739871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38992 (local address 127.0.97.60:46813)
0620 03:04:40.740109 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.740115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.740673 (+ 558us) server_negotiation.cc:407] Connection header received
0620 03:04:40.740890 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.740899 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.741008 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.741144 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:40.742059 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38992: BlockingRecv error: recv got EOF from 127.0.0.1:38992 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 03:04:40.761953 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:40.754320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.0.97.61:39697)
0620 03:04:40.754544 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.754550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.755090 (+ 540us) server_negotiation.cc:407] Connection header received
0620 03:04:40.755319 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.755325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.755419 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.755523 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:04:40.756039 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.757182 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.759205 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.760100 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.760459 (+ 359us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:40.761075 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:04:40.761211 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:04:40.761303 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:40.761474 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 03:04:40.761553 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 03:04:40.771510 23398 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:40.769743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39002 (local address 127.0.97.60:46813)
0620 03:04:40.769940 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.769946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.769968 (+ 22us) server_negotiation.cc:407] Connection header received
0620 03:04:40.770247 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.770253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.770356 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.770476 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:40.771295 (+ 819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39002: BlockingRecv error: recv got EOF from 127.0.0.1:39002 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 03:04:40.798868 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:40.791519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43930 (local address 127.0.97.61:39697)
0620 03:04:40.791730 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.791739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.792241 (+ 502us) server_negotiation.cc:407] Connection header received
0620 03:04:40.792466 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.792477 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.792576 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.792716 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:04:40.793165 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.794485 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.796619 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.797323 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.797453 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:40.798030 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:04:40.798166 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:04:40.798256 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:40.798424 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 03:04:40.798518 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 03:04:40.808279 23398 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:40.805974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39010 (local address 127.0.97.60:46813)
0620 03:04:40.806151 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.806157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.806736 (+ 579us) server_negotiation.cc:407] Connection header received
0620 03:04:40.806971 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.806976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.807073 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.807196 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:40.808039 (+ 843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39010: BlockingRecv error: recv got EOF from 127.0.0.1:39010 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 03:04:40.850237 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:40.843824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43934 (local address 127.0.97.61:39697)
0620 03:04:40.844053 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.844060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.844490 (+ 430us) server_negotiation.cc:407] Connection header received
0620 03:04:40.844679 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.844685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.844792 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.844937 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:04:40.845369 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.846677 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.848214 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.848870 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.849019 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:40.849377 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:04:40.849512 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:04:40.849687 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:40.849803 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 03:04:40.849887 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260620 03:04:40.858342 23398 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:40.856331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39012 (local address 127.0.97.60:46813)
0620 03:04:40.856542 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.856548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.856847 (+ 299us) server_negotiation.cc:407] Connection header received
0620 03:04:40.856996 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.857002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.857112 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.857242 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:40.858083 (+ 841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39012: BlockingRecv error: recv got EOF from 127.0.0.1:39012 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 03:04:40.931588 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:40.925707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.0.97.61:39697)
0620 03:04:40.925896 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.925903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.926286 (+ 383us) server_negotiation.cc:407] Connection header received
0620 03:04:40.926453 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.926459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.926558 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.926672 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:04:40.927093 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.928291 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.929794 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:40.930417 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:40.930545 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:40.930890 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:04:40.931014 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:04:40.931100 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:40.931222 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 03:04:40.931298 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 03:04:40.940176 23398 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 03:04:40.937853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.0.97.60:46813)
0620 03:04:40.938022 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:40.938030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:40.938562 (+ 532us) server_negotiation.cc:407] Connection header received
0620 03:04:40.938780 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:40.938786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:40.938874 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:40.938949 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 03:04:40.939951 (+ 1002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39026: BlockingRecv error: recv got EOF from 127.0.0.1:39026 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 03:04:41.054630 23373 catalog_manager.cc:1259] Loaded cluster ID: feab58c59b3440e5aecf3865ac62a7c4
I20260620 03:04:41.054731 23373 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126: loading cluster ID for follower catalog manager: success
I20260620 03:04:41.057166 23373 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126: acquiring CA information for follower catalog manager: success
I20260620 03:04:41.058931 23373 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c27734695f94408b89668adaabf0126: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 03:04:41.083252 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:41.078515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43948 (local address 127.0.97.61:39697)
0620 03:04:41.078707 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:41.078712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:41.079060 (+ 348us) server_negotiation.cc:407] Connection header received
0620 03:04:41.079231 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:41.079236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:41.079310 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:41.079402 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:04:41.079795 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:41.080498 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:41.081847 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:41.082241 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:41.082336 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:41.082683 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:04:41.082808 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:04:41.082873 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:41.082994 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 03:04:41.083054 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 03:04:41.094518 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:41.088886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.0.97.60:46813)
0620 03:04:41.089117 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:41.089123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:41.089299 (+ 176us) server_negotiation.cc:407] Connection header received
0620 03:04:41.089450 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:41.089454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:41.089691 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:41.089810 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:04:41.090160 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:41.090888 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:41.092214 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:41.092692 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:41.092792 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:41.093256 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:04:41.093936 (+ 680us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:04:41.094057 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:41.094122 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 03:04:41.094191 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 03:04:41.119221 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:41.101033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39030 (local address 127.0.97.60:46813)
0620 03:04:41.101292 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:41.101300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:41.113149 (+ 11849us) server_negotiation.cc:407] Connection header received
0620 03:04:41.113432 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:41.113439 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:41.113545 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:41.113690 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:04:41.114344 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:41.115242 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:41.117028 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:41.117698 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:41.117842 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:41.118362 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:04:41.118542 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:04:41.118592 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:41.118839 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 03:04:41.118927 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 03:04:41.119277 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:41.100957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43962 (local address 127.0.97.61:39697)
0620 03:04:41.101224 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:41.101233 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:41.112874 (+ 11641us) server_negotiation.cc:407] Connection header received
0620 03:04:41.113171 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:41.113179 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:41.113270 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:41.113338 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:04:41.114242 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:41.115082 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:41.117328 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:41.117838 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:41.117961 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:41.118532 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:04:41.118650 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:04:41.118737 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:41.118918 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 03:04:41.119000 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 03:04:41.119642 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 03:04:41.100618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.0.97.62:38693)
0620 03:04:41.111178 (+ 10560us) server_negotiation.cc:206] Beginning negotiation
0620 03:04:41.111187 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 03:04:41.112050 (+ 863us) server_negotiation.cc:407] Connection header received
0620 03:04:41.112467 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 03:04:41.112477 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 03:04:41.112769 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 03:04:41.112980 (+ 211us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 03:04:41.113609 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:41.114772 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:41.117169 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 03:04:41.117852 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 03:04:41.117962 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 03:04:41.118484 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 03:04:41.118940 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 03:04:41.119035 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 03:04:41.119237 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 03:04:41.119332 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10370,"thread_start_us":10573,"threads_started":1}
I20260620 03:04:41.124400 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 23141
I20260620 03:04:41.125614 23203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:04:41.237048 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 23141
W20260620 03:04:41.258517 23308 connection.cc:535] server connection from 127.0.0.1:38932 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 03:04:41.258922 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 23215
I20260620 03:04:41.260078 23277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:04:41.372706 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 23215
I20260620 03:04:41.392942 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRaC98/build/asan/bin/kudu pid 23289
I20260620 03:04:41.394105 23351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 03:04:41.513466 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRaC98/build/asan/bin/kudu with pid 23289
2026-06-20T03:04:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4389 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9296 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (361118 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST